2023-12-19T18:25:59.807 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-19T18:25:59.807 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270 --verbose --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270/orig.config.yaml 2023-12-19T18:25:59.862 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T18:25:59.963 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270 branch: wip-yuri8-testing-2023-12-15-0911 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7497270' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-12-15-0911 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-12-15-0911 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2960 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri8-testing-2023-12-15-0911 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3NlDAzADjs2CAiF8SUiU1CqJkoIMMlo5ctm5kltw245hoHt/gmD5NzfMizrAUe/wvfzqDuHwqvjsPPp65bHXg= smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALVpFu+Wjmuv9JQ18CMiNiRBsCf+d32vdJbhBhrXfZ4KCETRVOChoasgP7fFX4R12DLFOymDH8TxuruOPLGhx4= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/yuriw/src/github.com_ceph_ceph-c_wip-yuri8-testing-2023-12-15-0911/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-19_15:49:15 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-19T18:25:59.963 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa; will attempt to use it 2023-12-19T18:25:59.964 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks 2023-12-19T18:25:59.964 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-19T18:25:59.966 INFO:teuthology.task.internal:Checking packages... 2023-12-19T18:25:59.997 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c7586e95519b7be76f3b6e3a4b333888baf46bc9' 2023-12-19T18:25:59.997 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T18:25:59.997 INFO:teuthology.packaging:ref: None 2023-12-19T18:25:59.997 INFO:teuthology.packaging:tag: None 2023-12-19T18:25:59.998 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-12-15-0911 2023-12-19T18:25:59.998 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:25:59.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2023-12-15-0911 2023-12-19T18:26:00.150 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-202.gc7586e95 2023-12-19T18:26:00.151 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-19T18:26:00.162 INFO:teuthology.task.internal:no buildpackages task found 2023-12-19T18:26:00.162 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-19T18:26:00.224 INFO:teuthology.task.internal:Saving configuration 2023-12-19T18:26:00.243 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-19T18:26:00.254 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T18:26:00.280 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 18:20:57.626385', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3NlDAzADjs2CAiF8SUiU1CqJkoIMMlo5ctm5kltw245hoHt/gmD5NzfMizrAUe/wvfzqDuHwqvjsPPp65bHXg='} 2023-12-19T18:26:00.304 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 18:20:57.627828', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALVpFu+Wjmuv9JQ18CMiNiRBsCf+d32vdJbhBhrXfZ4KCETRVOChoasgP7fFX4R12DLFOymDH8TxuruOPLGhx4='} 2023-12-19T18:26:00.305 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-19T18:26:00.316 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-19T18:26:00.316 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-19T18:26:00.316 INFO:teuthology.run_tasks:Running task console_log... 2023-12-19T18:26:00.389 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5157897e50>, signals=[15]) 2023-12-19T18:26:00.390 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-19T18:26:00.399 INFO:teuthology.task.internal:Opening connections... 2023-12-19T18:26:00.399 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:26:00.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:26:00.500 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2023-12-19T18:26:00.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:26:00.584 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-19T18:26:00.592 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2023-12-19T18:26:00.611 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2023-12-19T18:26:00.611 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2023-12-19T18:26:00.669 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2023-12-19T18:26:00.669 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="9" 2023-12-19T18:26:00.669 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2023-12-19T18:26:00.669 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2023-12-19T18:26:00.669 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="9" 2023-12-19T18:26:00.670 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el9" 2023-12-19T18:26:00.670 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-19T18:26:00.670 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2023-12-19T18:26:00.670 INFO:teuthology.orchestra.run.smithi059.stdout:LOGO="fedora-logo-icon" 2023-12-19T18:26:00.670 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-19T18:26:00.670 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2023-12-19T18:26:00.670 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-19T18:26:00.670 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-19T18:26:00.670 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-19T18:26:00.671 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2023-12-19T18:26:00.689 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2023-12-19T18:26:00.706 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2023-12-19T18:26:00.706 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2023-12-19T18:26:00.762 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2023-12-19T18:26:00.762 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-19T18:26:00.763 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-19T18:26:00.763 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2023-12-19T18:26:00.788 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-19T18:26:00.800 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-19T18:26:00.812 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-19T18:26:00.812 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2023-12-19T18:26:00.814 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2023-12-19T18:26:00.830 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-19T18:26:00.841 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-19T18:26:00.841 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2023-12-19T18:26:00.872 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2023-12-19T18:26:00.895 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-19T18:26:00.955 INFO:teuthology.run_tasks:Running task kernel... 2023-12-19T18:26:00.972 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-19T18:26:00.972 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-19T18:26:00.972 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-19T18:26:00.973 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-19T18:26:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:26:00.988 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2023-12-19T18:26:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:26:00.989 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2023-12-19T18:26:01.043 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-361.el9.x86_64 2023-12-19T18:26:01.044 INFO:teuthology.task.kernel:Running kernel on smithi089: 5.14.0-361.el9.x86_64 2023-12-19T18:26:01.044 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2023-12-19T18:26:01.047 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-361.el9.x86_64 2023-12-19T18:26:01.047 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.14.0-361.el9.x86_64 2023-12-19T18:26:01.047 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2023-12-19T18:26:03.098 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-19T18:26:03.144 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-12-19T18:26:04.490 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2023-12-19T18:26:04.745 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2023-12-19T18:26:06.144 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 7.4 MB/s | 7.9 MB 00:01 2023-12-19T18:26:07.605 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 7.9 MB 00:02 2023-12-19T18:26:08.978 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-12-19T18:26:13.778 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.8 MB 00:01 2023-12-19T18:26:15.321 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-12-19T18:26:17.130 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2023-12-19T18:26:21.655 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 18 MB 00:12 2023-12-19T18:26:21.739 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-12-19T18:26:23.937 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-19T18:26:23.937 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-19T18:26:24.036 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:26:24.042 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:26:24.042 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2023-12-19T18:26:24.042 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:26:24.042 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:26:24.042 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-19T18:26:24.042 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-12-19T18:26:24.042 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-19T18:26:24.043 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-19T18:26:24.043 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-19T18:26:24.043 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:26:24.043 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:26:24.043 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:26:24.043 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2023-12-19T18:26:24.043 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:26:24.043 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 96 M 2023-12-19T18:26:24.044 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 121 M 2023-12-19T18:26:24.044 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:26:24.524 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 14 MB/s | 5.6 MB 00:00 2023-12-19T18:26:25.007 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 23 MB/s | 20 MB 00:00 2023-12-19T18:26:25.624 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): kernel-modules-core-5.14.0-391.el9.x86_6 29 MB/s | 32 MB 00:01 2023-12-19T18:26:25.991 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 20 MB/s | 38 MB 00:01 2023-12-19T18:26:25.991 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:26:25.991 INFO:teuthology.orchestra.run.smithi089.stdout:Total 49 MB/s | 96 MB 00:01 2023-12-19T18:26:26.106 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 5.8 MB 00:00 2023-12-19T18:26:26.843 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:26:27.225 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:26:27.225 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:26:27.515 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 15 kB 00:00 2023-12-19T18:26:28.550 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:26:28.550 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:26:29.316 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2023-12-19T18:26:31.198 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:26:31.845 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-19T18:26:32.008 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:26:34.266 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-12-19T18:26:34.730 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:26:34.912 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:26:36.490 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-19T18:26:36.490 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-19T18:26:36.581 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:26:36.585 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:26:36.585 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-12-19T18:26:36.585 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:26:36.585 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout:Install 4 Packages 2023-12-19T18:26:36.586 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:26:36.588 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 96 M 2023-12-19T18:26:36.588 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 121 M 2023-12-19T18:26:36.589 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:26:37.093 INFO:teuthology.orchestra.run.smithi059.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 15 MB/s | 5.6 MB 00:00 2023-12-19T18:26:37.426 INFO:teuthology.orchestra.run.smithi059.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 29 MB/s | 20 MB 00:00 2023-12-19T18:26:39.541 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:26:39.653 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:26:40.893 INFO:teuthology.orchestra.run.smithi059.stdout:(3/4): kernel-modules-core-5.14.0-391.el9.x86_6 8.5 MB/s | 32 MB 00:03 2023-12-19T18:26:44.282 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:26:46.526 INFO:teuthology.orchestra.run.smithi059.stdout:(4/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 3.9 MB/s | 38 MB 00:09 2023-12-19T18:26:46.526 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:26:46.527 INFO:teuthology.orchestra.run.smithi059.stdout:Total 9.7 MB/s | 96 MB 00:09 2023-12-19T18:26:47.418 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:26:47.848 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:26:47.849 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:26:49.203 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:26:49.203 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:26:51.985 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:26:52.661 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-19T18:26:52.784 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:26:55.669 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:26:55.837 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:27:00.828 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:27:00.972 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:05.995 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:13.158 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:13.202 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:13.562 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-19T18:27:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:27:13.563 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:27:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:27:13.970 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:27:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-19T18:27:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-19T18:27:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-19T18:27:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-19T18:27:13.970 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:27:13.970 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:27:14.481 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T18:27:14.622 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T18:27:14.624 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T18:27:14.624 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-19T18:27:14.624 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-19T18:27:14.624 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-19T18:27:14.624 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-19T18:27:14.624 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi089.front.sepia.ceph.com, path=None, version=distro) 2023-12-19T18:27:14.624 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T18:27:14.742 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T18:27:14.742 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T18:27:14.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -qi grub2-tools 2023-12-19T18:27:14.776 INFO:teuthology.orchestra.run.smithi089.stdout:Name : grub2-tools 2023-12-19T18:27:14.776 INFO:teuthology.orchestra.run.smithi089.stdout:Epoch : 1 2023-12-19T18:27:14.776 INFO:teuthology.orchestra.run.smithi089.stdout:Version : 2.06 2023-12-19T18:27:14.776 INFO:teuthology.orchestra.run.smithi089.stdout:Release : 68.el9 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Architecture: x86_64 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Group : Unspecified 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Size : 8264254 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:License : GPLv3+ 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Packager : builder@centos.org 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Vendor : CentOS 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-19T18:27:14.777 INFO:teuthology.orchestra.run.smithi089.stdout:Summary : Support tools for GRUB. 2023-12-19T18:27:14.778 INFO:teuthology.orchestra.run.smithi089.stdout:Description : 2023-12-19T18:27:14.778 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:27:14.778 INFO:teuthology.orchestra.run.smithi089.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-19T18:27:14.778 INFO:teuthology.orchestra.run.smithi089.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-19T18:27:14.778 INFO:teuthology.orchestra.run.smithi089.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-19T18:27:14.778 INFO:teuthology.orchestra.run.smithi089.stdout:hardware devices. 2023-12-19T18:27:14.778 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:27:14.778 INFO:teuthology.orchestra.run.smithi089.stdout:This subpackage provides tools for support of all platforms. 2023-12-19T18:27:14.779 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-19T18:27:14.779 INFO:teuthology.task.kernel:Updating grub on smithi089 to boot 5.14.0-391.el9.x86_64 2023-12-19T18:27:14.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-19T18:27:15.430 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2023-12-19T18:27:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-19T18:27:16.785 INFO:teuthology.orchestra.run.smithi089.stderr:done 2023-12-19T18:27:16.788 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2023-12-19T18:27:16.803 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.pHBYlzrfbm 2023-12-19T18:27:16.804 DEBUG:teuthology.orchestra.run.smithi089:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pHBYlzrfbm 2023-12-19T18:27:16.873 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.pHBYlzrfbm 2023-12-19T18:27:17.012 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.pHBYlzrfbm is 6KB 2023-12-19T18:27:17.062 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.pHBYlzrfbm 2023-12-19T18:27:17.077 DEBUG:teuthology.orchestra.run.smithi089:> sudo /bin/ls /boot/loader/entries || true 2023-12-19T18:27:17.152 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-19T18:27:17.152 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-19T18:27:17.152 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-19T18:27:17.152 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-19T18:27:17.153 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-19T18:27:17.301 DEBUG:teuthology.orchestra.run.smithi089:> sudo shutdown -r now 2023-12-19T18:27:36.584 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:36.628 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-19T18:27:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:27:37.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:27:37.454 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:27:37.454 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:27:37.454 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:27:37.454 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-19T18:27:37.454 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-19T18:27:37.454 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-19T18:27:37.454 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-19T18:27:37.454 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:27:37.454 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:27:37.959 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T18:27:38.100 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T18:27:38.100 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T18:27:38.100 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-19T18:27:38.101 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-19T18:27:38.101 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-19T18:27:38.101 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-19T18:27:38.101 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2023-12-19T18:27:38.101 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T18:27:38.224 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T18:27:38.224 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T18:27:38.224 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2023-12-19T18:27:38.267 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.06 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 68.el9 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Group : Unspecified 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 8264254 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : builder@centos.org 2023-12-19T18:27:38.268 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:27:38.269 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2023-12-19T18:27:38.270 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-19T18:27:38.270 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 5.14.0-391.el9.x86_64 2023-12-19T18:27:38.270 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-19T18:27:38.908 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2023-12-19T18:27:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-19T18:27:40.379 INFO:teuthology.orchestra.run.smithi059.stderr:done 2023-12-19T18:27:40.382 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2023-12-19T18:27:40.397 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.Hdue0eyt4R 2023-12-19T18:27:40.397 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Hdue0eyt4R 2023-12-19T18:27:40.470 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.Hdue0eyt4R 2023-12-19T18:27:40.612 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.Hdue0eyt4R is 6KB 2023-12-19T18:27:40.660 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.Hdue0eyt4R 2023-12-19T18:27:40.675 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2023-12-19T18:27:40.744 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-19T18:27:40.744 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-19T18:27:40.744 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-19T18:27:40.745 INFO:teuthology.orchestra.run.smithi059.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-19T18:27:40.746 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-19T18:27:40.894 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2023-12-19T18:27:47.313 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-19T18:27:47.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T18:27:47.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:28:05.778 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T18:28:10.901 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-19T18:28:10.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-19T18:28:10.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:28:14.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T18:28:14.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:28:17.841 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T18:28:29.330 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2023-12-19T18:28:29.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T18:28:29.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:28:38.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-19T18:28:38.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:28:48.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T18:29:03.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T18:29:03.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:29:03.635 DEBUG:teuthology.orchestra.run.smithi089:> true 2023-12-19T18:29:04.003 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T18:29:04.003 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-391.el9.x86_64"... 2023-12-19T18:29:04.003 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2023-12-19T18:29:04.017 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-391.el9.x86_64 2023-12-19T18:29:04.018 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-19T18:29:04.018 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-19T18:29:04.018 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-19T18:29:05.018 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-19T18:29:05.019 DEBUG:teuthology.orchestra.run.smithi089:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-19T18:29:05.108 INFO:teuthology.orchestra.run.smithi089.stdout:ttyS1 2023-12-19T18:29:05.158 DEBUG:teuthology.parallel:result is None 2023-12-19T18:29:38.341 DEBUG:teuthology.orchestra.remote:timed out 2023-12-19T18:29:50.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-19T18:29:50.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:29:50.708 DEBUG:teuthology.orchestra.run.smithi059:> true 2023-12-19T18:29:51.066 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-19T18:29:51.066 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-391.el9.x86_64"... 2023-12-19T18:29:51.066 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2023-12-19T18:29:51.081 INFO:teuthology.orchestra.run.smithi059.stdout:5.14.0-391.el9.x86_64 2023-12-19T18:29:51.082 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-19T18:29:51.082 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-19T18:29:51.082 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-19T18:29:52.082 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-19T18:29:52.083 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-19T18:29:52.153 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2023-12-19T18:29:52.196 DEBUG:teuthology.parallel:result is None 2023-12-19T18:29:52.196 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-19T18:29:52.204 INFO:teuthology.task.internal:Creating test directory... 2023-12-19T18:29:52.204 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-19T18:29:52.207 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-19T18:29:52.225 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-19T18:29:52.232 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-19T18:29:52.242 INFO:teuthology.task.internal:Creating archive directory... 2023-12-19T18:29:52.243 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-19T18:29:52.265 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-19T18:29:52.296 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-19T18:29:52.303 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-19T18:29:52.303 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-19T18:29:52.328 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-19T18:29:52.365 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T18:29:52.374 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T18:29:52.382 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T18:29:52.387 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T18:29:52.389 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-19T18:29:52.397 INFO:teuthology.task.internal:Configuring sudo... 2023-12-19T18:29:52.397 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-19T18:29:52.427 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-19T18:29:52.459 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-19T18:29:52.468 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-19T18:29:52.468 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-19T18:29:52.501 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-19T18:29:52.517 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T18:29:52.596 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T18:29:52.660 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T18:29:52.704 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T18:29:52.749 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:29:52.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-19T18:29:52.821 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T18:29:52.865 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T18:29:52.922 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T18:29:53.006 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T18:29:53.052 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:29:53.052 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-19T18:29:53.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2023-12-19T18:29:53.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2023-12-19T18:29:53.174 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-19T18:29:53.231 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-19T18:29:53.532 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-19T18:29:53.541 INFO:teuthology.task.internal:Starting timer... 2023-12-19T18:29:53.541 INFO:teuthology.run_tasks:Running task pcp... 2023-12-19T18:29:53.557 INFO:teuthology.run_tasks:Running task selinux... 2023-12-19T18:29:53.572 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-19T18:29:53.572 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2023-12-19T18:29:53.639 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2023-12-19T18:29:53.641 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2023-12-19T18:29:53.688 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2023-12-19T18:29:53.690 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-19T18:29:53.690 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2023-12-19T18:29:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2023-12-19T18:29:53.712 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2023-12-19T18:29:53.737 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2023-12-19T18:29:53.737 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi089.front.sepia.ceph.com': 'permissive'} 2023-12-19T18:29:53.737 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-19T18:29:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:29:53.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-19T18:29:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:29:53.813 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-19T18:29:53.813 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2023-12-19T18:29:53.864 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2023-12-19T18:29:53.895 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-19T18:29:53.906 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-19T18:29:53.906 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-19T18:29:53.925 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-19T18:29:53.926 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi089.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-19T18:35:36.058 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi089.front.sepia.ceph.com')] 2023-12-19T18:35:36.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-19T18:35:36.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:35:36.157 DEBUG:teuthology.orchestra.run.smithi059:> true 2023-12-19T18:35:36.183 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-19T18:35:36.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T18:35:36.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:35:36.268 DEBUG:teuthology.orchestra.run.smithi089:> true 2023-12-19T18:35:36.337 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T18:35:36.337 INFO:teuthology.run_tasks:Running task clock... 2023-12-19T18:35:36.349 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-19T18:35:36.349 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-19T18:35:36.350 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T18:35:36.352 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-19T18:35:36.352 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T18:35:36.385 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-19T18:35:36.406 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-19T18:35:36.409 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-19T18:35:36.427 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-19T18:35:36.442 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2023-12-19T18:35:36.457 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2023-12-19T18:35:36.459 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2023-12-19T18:35:36.469 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2023-12-19T18:35:36.480 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-19T18:35:36.485 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-19T18:35:36.500 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-19T18:35:36.501 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-19T18:35:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2023-12-19T18:35:36.547 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-19T18:35:36.547 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2023-12-19T18:35:36.547 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T18:35:36.547 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T18:35:36.547 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T18:35:36.548 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T18:35:36.548 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2023-12-19T18:35:36.552 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-19T18:35:36.552 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2023-12-19T18:35:36.552 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T18:35:36.552 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T18:35:36.552 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T18:35:36.552 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T18:35:36.552 INFO:teuthology.run_tasks:Running task install... 2023-12-19T18:35:36.562 DEBUG:teuthology.task.install:project ceph 2023-12-19T18:35:36.562 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}} 2023-12-19T18:35:36.563 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-19T18:35:36.563 INFO:teuthology.task.install:Using flavor: default 2023-12-19T18:35:36.568 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-19T18:35:36.568 INFO:teuthology.task.install:extra packages: [] 2023-12-19T18:35:36.569 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-19T18:35:36.569 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T18:35:36.570 INFO:teuthology.packaging:ref: None 2023-12-19T18:35:36.570 INFO:teuthology.packaging:tag: None 2023-12-19T18:35:36.570 INFO:teuthology.packaging:branch: quincy 2023-12-19T18:35:36.570 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:35:36.570 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-19T18:35:36.572 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-19T18:35:36.572 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T18:35:36.572 INFO:teuthology.packaging:ref: None 2023-12-19T18:35:36.572 INFO:teuthology.packaging:tag: None 2023-12-19T18:35:36.572 INFO:teuthology.packaging:branch: quincy 2023-12-19T18:35:36.572 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:35:36.573 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-19T18:35:36.790 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/ 2023-12-19T18:35:36.790 INFO:teuthology.task.install.rpm:Package version is 17.2.7-131.ga88f3edf 2023-12-19T18:35:36.877 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/ 2023-12-19T18:35:36.877 INFO:teuthology.task.install.rpm:Package version is 17.2.7-131.ga88f3edf 2023-12-19T18:35:36.960 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-19T18:35:36.960 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:35:36.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-19T18:35:36.988 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-19T18:35:36.988 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T18:35:36.988 INFO:teuthology.packaging:ref: None 2023-12-19T18:35:36.988 INFO:teuthology.packaging:tag: None 2023-12-19T18:35:36.989 INFO:teuthology.packaging:branch: quincy 2023-12-19T18:35:36.989 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:35:36.989 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-19T18:35:37.045 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-19T18:35:37.045 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:35:37.045 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-19T18:35:37.056 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-19T18:35:37.080 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-19T18:35:37.081 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T18:35:37.081 INFO:teuthology.packaging:ref: None 2023-12-19T18:35:37.081 INFO:teuthology.packaging:tag: None 2023-12-19T18:35:37.081 INFO:teuthology.packaging:branch: quincy 2023-12-19T18:35:37.081 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:35:37.081 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-19T18:35:37.136 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-19T18:35:37.154 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-19T18:35:37.202 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2023-12-19T18:35:37.204 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2023-12-19T18:35:37.244 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-19T18:35:37.276 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2023-12-19T18:35:37.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2023-12-19T18:35:37.443 INFO:teuthology.orchestra.run.smithi089.stdout:73 files removed 2023-12-19T18:35:37.470 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw 2023-12-19T18:35:37.542 INFO:teuthology.orchestra.run.smithi059.stdout:73 files removed 2023-12-19T18:35:37.568 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2023-12-19T18:35:38.125 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 227 kB/s | 77 kB 00:00 2023-12-19T18:35:38.250 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 225 kB/s | 77 kB 00:00 2023-12-19T18:35:38.509 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 27 kB/s | 9.1 kB 00:00 2023-12-19T18:35:38.667 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 25 kB/s | 9.1 kB 00:00 2023-12-19T18:35:38.910 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 4.9 kB/s | 1.8 kB 00:00 2023-12-19T18:35:39.093 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 4.6 kB/s | 1.8 kB 00:00 2023-12-19T18:35:39.312 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-12-19T18:35:39.537 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-12-19T18:35:40.788 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2023-12-19T18:35:41.145 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2023-12-19T18:35:42.987 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 7.9 MB 00:01 2023-12-19T18:35:43.061 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 7.9 MB 00:01 2023-12-19T18:35:46.570 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 18 MB 00:02 2023-12-19T18:35:51.962 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 18 MB 00:07 2023-12-19T18:35:52.313 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 5.8 MB 00:02 2023-12-19T18:35:53.938 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-12-19T18:35:54.876 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 24 MB/s | 20 MB 00:00 2023-12-19T18:35:57.105 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 5.8 MB 00:01 2023-12-19T18:35:58.889 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2023-12-19T18:35:59.793 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-12-19T18:36:00.874 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-12-19T18:36:02.374 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:36:02.377 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:02.377 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-19T18:36:02.377 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:02.377 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 14 M 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 3.1 M 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.9 M 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 5.0 M 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 23 M 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 24 k 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 635 k 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-19T18:36:02.378 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 457 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 4.9 M 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 45 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 116 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 156 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 319 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 292 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 99 k 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:36:02.379 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:02.380 INFO:teuthology.orchestra.run.smithi059.stdout:Install 22 Packages 2023-12-19T18:36:02.380 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2023-12-19T18:36:02.380 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:02.380 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 58 M 2023-12-19T18:36:02.380 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:36:03.619 INFO:teuthology.orchestra.run.smithi059.stdout:(1/24): ceph-radosgw-17.2.7-131.ga88f3edf.el9.x 13 MB/s | 14 MB 00:01 2023-12-19T18:36:03.694 INFO:teuthology.orchestra.run.smithi059.stdout:(2/24): ceph-base-17.2.7-131.ga88f3edf.el9.x86_ 4.4 MB/s | 5.0 MB 00:01 2023-12-19T18:36:03.720 INFO:teuthology.orchestra.run.smithi059.stdout:(3/24): ceph-selinux-17.2.7-131.ga88f3edf.el9.x 240 kB/s | 24 kB 00:00 2023-12-19T18:36:03.786 INFO:teuthology.orchestra.run.smithi059.stdout:(4/24): libcephfs2-17.2.7-131.ga88f3edf.el9.x86 6.8 MB/s | 635 kB 00:00 2023-12-19T18:36:03.828 INFO:teuthology.orchestra.run.smithi059.stdout:(5/24): libradosstriper1-17.2.7-131.ga88f3edf.e 4.1 MB/s | 457 kB 00:00 2023-12-19T18:36:03.962 INFO:teuthology.orchestra.run.smithi059.stdout:(6/24): python3-ceph-argparse-17.2.7-131.ga88f3 338 kB/s | 45 kB 00:00 2023-12-19T18:36:04.054 INFO:teuthology.orchestra.run.smithi059.stdout:(7/24): librgw2-17.2.7-131.ga88f3edf.el9.x86_64 18 MB/s | 4.9 MB 00:00 2023-12-19T18:36:04.079 INFO:teuthology.orchestra.run.smithi059.stdout:(8/24): python3-ceph-common-17.2.7-131.ga88f3ed 994 kB/s | 116 kB 00:00 2023-12-19T18:36:04.104 INFO:teuthology.orchestra.run.smithi059.stdout:(9/24): python3-cephfs-17.2.7-131.ga88f3edf.el9 3.0 MB/s | 156 kB 00:00 2023-12-19T18:36:04.180 INFO:teuthology.orchestra.run.smithi059.stdout:(10/24): python3-rados-17.2.7-131.ga88f3edf.el9 3.1 MB/s | 319 kB 00:00 2023-12-19T18:36:04.206 INFO:teuthology.orchestra.run.smithi059.stdout:(11/24): python3-rbd-17.2.7-131.ga88f3edf.el9.x 2.9 MB/s | 292 kB 00:00 2023-12-19T18:36:04.263 INFO:teuthology.orchestra.run.smithi059.stdout:(12/24): python3-rgw-17.2.7-131.ga88f3edf.el9.x 1.2 MB/s | 99 kB 00:00 2023-12-19T18:36:04.330 INFO:teuthology.orchestra.run.smithi059.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2023-12-19T18:36:04.347 INFO:teuthology.orchestra.run.smithi059.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 537 kB/s | 44 kB 00:00 2023-12-19T18:36:04.398 INFO:teuthology.orchestra.run.smithi059.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-12-19T18:36:04.423 INFO:teuthology.orchestra.run.smithi059.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-12-19T18:36:04.440 INFO:teuthology.orchestra.run.smithi059.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 998 kB/s | 41 kB 00:00 2023-12-19T18:36:04.765 INFO:teuthology.orchestra.run.smithi059.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 149 kB/s | 49 kB 00:00 2023-12-19T18:36:04.815 INFO:teuthology.orchestra.run.smithi059.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 788 kB/s | 309 kB 00:00 2023-12-19T18:36:04.909 INFO:teuthology.orchestra.run.smithi059.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 475 kB/s | 67 kB 00:00 2023-12-19T18:36:05.041 INFO:teuthology.orchestra.run.smithi059.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2023-12-19T18:36:05.116 INFO:teuthology.orchestra.run.smithi059.stdout:(22/24): librados2-17.2.7-131.ga88f3edf.el9.x86 15 MB/s | 3.1 MB 00:00 2023-12-19T18:36:05.416 INFO:teuthology.orchestra.run.smithi059.stdout:(23/24): librbd1-17.2.7-131.ga88f3edf.el9.x86_6 7.8 MB/s | 2.9 MB 00:00 2023-12-19T18:36:05.607 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-12-19T18:36:06.317 INFO:teuthology.orchestra.run.smithi059.stdout:(24/24): ceph-common-17.2.7-131.ga88f3edf.el9.x 6.0 MB/s | 23 MB 00:03 2023-12-19T18:36:06.320 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:36:06.321 INFO:teuthology.orchestra.run.smithi059.stdout:Total 15 MB/s | 58 MB 00:03 2023-12-19T18:36:06.321 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:36:06.360 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:36:06.360 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:36:06.866 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:36:06.866 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:36:08.083 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:36:08.094 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:36:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-12-19T18:36:08.094 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:36:08.094 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:36:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 14 M 2023-12-19T18:36:08.094 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2023-12-19T18:36:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 3.1 M 2023-12-19T18:36:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.9 M 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 5.0 M 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 23 M 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 24 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 635 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 457 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 4.9 M 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-19T18:36:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 45 k 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 116 k 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 156 k 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 319 k 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 292 k 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 99 k 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout:Install 22 Packages 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 2 Packages 2023-12-19T18:36:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:08.097 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 58 M 2023-12-19T18:36:08.097 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:36:08.132 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:36:08.497 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2023-12-19T18:36:08.552 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T18:36:08.645 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T18:36:08.725 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 3/26 2023-12-19T18:36:08.753 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T18:36:08.826 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T18:36:08.889 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2023-12-19T18:36:08.925 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2023-12-19T18:36:09.055 INFO:teuthology.orchestra.run.smithi089.stdout:(1/24): ceph-base-17.2.7-131.ga88f3edf.el9.x86_ 5.5 MB/s | 5.0 MB 00:00 2023-12-19T18:36:09.122 INFO:teuthology.orchestra.run.smithi089.stdout:(2/24): ceph-selinux-17.2.7-131.ga88f3edf.el9.x 361 kB/s | 24 kB 00:00 2023-12-19T18:36:09.222 INFO:teuthology.orchestra.run.smithi089.stdout:(3/24): libcephfs2-17.2.7-131.ga88f3edf.el9.x86 6.2 MB/s | 635 kB 00:00 2023-12-19T18:36:09.323 INFO:teuthology.orchestra.run.smithi089.stdout:(4/24): libradosstriper1-17.2.7-131.ga88f3edf.e 4.5 MB/s | 457 kB 00:00 2023-12-19T18:36:09.382 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2023-12-19T18:36:09.406 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 8/26 2023-12-19T18:36:09.461 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 8/26 2023-12-19T18:36:09.490 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2023-12-19T18:36:09.533 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x 10/26 2023-12-19T18:36:09.570 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 11/26 2023-12-19T18:36:09.634 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 12/26 2023-12-19T18:36:09.659 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T18:36:09.725 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T18:36:09.765 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2023-12-19T18:36:09.801 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2023-12-19T18:36:09.833 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2023-12-19T18:36:10.056 INFO:teuthology.orchestra.run.smithi089.stdout:(5/24): ceph-radosgw-17.2.7-131.ga88f3edf.el9.x 7.4 MB/s | 14 MB 00:01 2023-12-19T18:36:10.131 INFO:teuthology.orchestra.run.smithi089.stdout:(6/24): librgw2-17.2.7-131.ga88f3edf.el9.x86_64 6.1 MB/s | 4.9 MB 00:00 2023-12-19T18:36:10.144 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2023-12-19T18:36:10.148 INFO:teuthology.orchestra.run.smithi089.stdout:(7/24): python3-ceph-argparse-17.2.7-131.ga88f3 492 kB/s | 45 kB 00:00 2023-12-19T18:36:10.171 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 18/26 2023-12-19T18:36:10.190 INFO:teuthology.orchestra.run.smithi089.stdout:(8/24): python3-ceph-common-17.2.7-131.ga88f3ed 1.9 MB/s | 116 kB 00:00 2023-12-19T18:36:10.216 INFO:teuthology.orchestra.run.smithi089.stdout:(9/24): python3-cephfs-17.2.7-131.ga88f3edf.el9 2.3 MB/s | 156 kB 00:00 2023-12-19T18:36:10.245 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 18/26 2023-12-19T18:36:10.282 INFO:teuthology.orchestra.run.smithi089.stdout:(10/24): python3-rados-17.2.7-131.ga88f3edf.el9 3.4 MB/s | 319 kB 00:00 2023-12-19T18:36:10.308 INFO:teuthology.orchestra.run.smithi089.stdout:(11/24): python3-rbd-17.2.7-131.ga88f3edf.el9.x 3.1 MB/s | 292 kB 00:00 2023-12-19T18:36:10.312 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 19/26 2023-12-19T18:36:10.340 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86 20/26 2023-12-19T18:36:10.525 INFO:teuthology.orchestra.run.smithi089.stdout:(12/24): ceph-common-17.2.7-131.ga88f3edf.el9.x 9.5 MB/s | 23 MB 00:02 2023-12-19T18:36:10.575 INFO:teuthology.orchestra.run.smithi089.stdout:(13/24): python3-rgw-17.2.7-131.ga88f3edf.el9.x 340 kB/s | 99 kB 00:00 2023-12-19T18:36:10.650 INFO:teuthology.orchestra.run.smithi089.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 465 kB/s | 159 kB 00:00 2023-12-19T18:36:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 328 kB/s | 44 kB 00:00 2023-12-19T18:36:10.709 INFO:teuthology.orchestra.run.smithi089.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2023-12-19T18:36:10.734 INFO:teuthology.orchestra.run.smithi089.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2023-12-19T18:36:10.760 INFO:teuthology.orchestra.run.smithi089.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 553 kB/s | 41 kB 00:00 2023-12-19T18:36:10.952 INFO:teuthology.orchestra.run.smithi089.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 224 kB/s | 49 kB 00:00 2023-12-19T18:36:10.968 INFO:teuthology.orchestra.run.smithi089.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 323 kB/s | 67 kB 00:00 2023-12-19T18:36:11.041 INFO:teuthology.orchestra.run.smithi089.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2023-12-19T18:36:11.286 INFO:teuthology.orchestra.run.smithi089.stdout:(22/24): librados2-17.2.7-131.ga88f3edf.el9.x86 9.7 MB/s | 3.1 MB 00:00 2023-12-19T18:36:11.411 INFO:teuthology.orchestra.run.smithi089.stdout:(23/24): librbd1-17.2.7-131.ga88f3edf.el9.x86_6 7.9 MB/s | 2.9 MB 00:00 2023-12-19T18:36:11.519 INFO:teuthology.orchestra.run.smithi089.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2023-12-19T18:36:11.526 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:36:11.526 INFO:teuthology.orchestra.run.smithi089.stdout:Total 17 MB/s | 58 MB 00:03 2023-12-19T18:36:11.527 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:36:11.555 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:36:11.555 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:36:12.024 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:36:12.024 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:36:12.870 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T18:36:12.912 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T18:36:13.231 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:36:13.476 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T18:36:13.515 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 22/26 2023-12-19T18:36:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2023-12-19T18:36:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 22/26 2023-12-19T18:36:13.583 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-19T18:36:13.583 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-19T18:36:13.583 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:13.616 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T18:36:13.688 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T18:36:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T18:36:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 3/26 2023-12-19T18:36:13.931 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T18:36:14.018 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T18:36:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2023-12-19T18:36:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2023-12-19T18:36:14.541 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2023-12-19T18:36:14.576 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 8/26 2023-12-19T18:36:14.622 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 8/26 2023-12-19T18:36:14.652 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2023-12-19T18:36:14.695 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x 10/26 2023-12-19T18:36:14.732 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 11/26 2023-12-19T18:36:14.794 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 12/26 2023-12-19T18:36:14.822 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T18:36:14.888 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T18:36:14.931 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2023-12-19T18:36:14.963 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2023-12-19T18:36:14.996 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2023-12-19T18:36:15.288 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2023-12-19T18:36:15.318 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 18/26 2023-12-19T18:36:15.407 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 18/26 2023-12-19T18:36:15.479 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 19/26 2023-12-19T18:36:15.503 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86 20/26 2023-12-19T18:36:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T18:36:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T18:36:18.525 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T18:36:18.561 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 22/26 2023-12-19T18:36:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 22/26 2023-12-19T18:36:18.637 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-19T18:36:18.637 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-19T18:36:18.637 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:18.679 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T18:36:23.410 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T18:36:23.411 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2023-12-19T18:36:23.411 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2023-12-19T18:36:23.411 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2023-12-19T18:36:23.411 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2023-12-19T18:36:23.411 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2023-12-19T18:36:23.411 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /root 2023-12-19T18:36:23.411 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2023-12-19T18:36:23.411 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:24.715 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 24/26 2023-12-19T18:36:24.792 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 24/26 2023-12-19T18:36:24.792 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:24.792 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-19T18:36:24.792 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T18:36:24.792 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T18:36:24.792 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:24.794 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-19T18:36:24.901 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-19T18:36:24.901 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 1/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 3/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 5/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 6/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 7/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x 8/26 2023-12-19T18:36:26.127 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86 9/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 10/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 11/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 12/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2023-12-19T18:36:26.128 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T18:36:26.129 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2023-12-19T18:36:26.129 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 25/26 2023-12-19T18:36:27.009 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T18:36:27.009 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:27.009 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2023-12-19T18:36:27.009 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.009 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.009 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:36:27.009 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.010 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.011 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.011 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-19T18:36:27.011 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.011 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.011 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:27.011 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-19T18:36:27.011 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:27.011 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:36:27.370 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2023-12-19T18:36:27.962 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:28 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:36:28.124 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:36:28.124 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:28.124 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-19T18:36:28.124 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:28.124 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:36:28.124 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 48 M 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 158 k 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2023-12-19T18:36:28.125 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:28.129 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 49 M 2023-12-19T18:36:28.129 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 195 M 2023-12-19T18:36:28.129 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:36:28.178 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T18:36:28.179 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2023-12-19T18:36:28.179 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2023-12-19T18:36:28.179 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2023-12-19T18:36:28.179 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2023-12-19T18:36:28.180 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2023-12-19T18:36:28.180 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2023-12-19T18:36:28.181 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2023-12-19T18:36:28.181 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:28.306 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 186 kB 00:00 2023-12-19T18:36:28.348 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-19T18:36:28.390 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-19T18:36:28.432 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-19T18:36:28.557 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): libcephsqlite-17.2.7-131.ga88f3edf.el9.x 368 kB/s | 158 kB 00:00 2023-12-19T18:36:29.459 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 24/26 2023-12-19T18:36:29.554 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 24/26 2023-12-19T18:36:29.554 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:29.554 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-19T18:36:29.554 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T18:36:29.555 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T18:36:29.555 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:29.556 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-19T18:36:29.629 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-19T18:36:29.629 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T18:36:30.825 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 1/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 3/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 5/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 6/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 7/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x 8/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86 9/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 10/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 11/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 12/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2023-12-19T18:36:30.826 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2023-12-19T18:36:30.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 25/26 2023-12-19T18:36:31.657 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): ceph-test-17.2.7-131.ga88f3edf.el9.x86_6 14 MB/s | 48 MB 00:03 2023-12-19T18:36:31.660 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:36:31.660 INFO:teuthology.orchestra.run.smithi059.stdout:Total 14 MB/s | 49 MB 00:03 2023-12-19T18:36:31.661 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:36:31.682 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:36:31.682 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:36:31.856 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-19T18:36:31.857 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-19T18:36:31.858 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-19T18:36:31.859 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:31.859 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:36:32.095 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:36:32.096 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:36:32.204 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-test 2023-12-19T18:36:32.721 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:36:32.756 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:27 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:36:32.811 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-19T18:36:32.853 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-19T18:36:32.881 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-19T18:36:32.910 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:36:32.911 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:36:32.911 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-12-19T18:36:32.911 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:36:32.911 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:36:32.911 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 48 M 2023-12-19T18:36:32.911 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 158 k 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2023-12-19T18:36:32.912 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:32.913 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 49 M 2023-12-19T18:36:32.913 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 195 M 2023-12-19T18:36:32.913 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:36:32.924 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-19T18:36:32.952 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 5/6 2023-12-19T18:36:33.076 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-12-19T18:36:33.143 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2023-12-19T18:36:33.193 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2023-12-19T18:36:33.227 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): libcephsqlite-17.2.7-131.ga88f3edf.el9.x 505 kB/s | 158 kB 00:00 2023-12-19T18:36:33.252 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2023-12-19T18:36:35.294 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): ceph-test-17.2.7-131.ga88f3edf.el9.x86_6 20 MB/s | 48 MB 00:02 2023-12-19T18:36:35.298 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:36:35.298 INFO:teuthology.orchestra.run.smithi089.stdout:Total 20 MB/s | 49 MB 00:02 2023-12-19T18:36:35.298 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:36:35.318 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:36:35.318 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:36:35.709 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:36:35.709 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:36:36.288 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:36:36.368 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-19T18:36:36.410 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-19T18:36:36.452 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-19T18:36:36.494 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-19T18:36:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 5/6 2023-12-19T18:36:37.630 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 5/6 2023-12-19T18:36:37.703 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 6/6 2023-12-19T18:36:39.728 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 6/6 2023-12-19T18:36:39.728 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 1/6 2023-12-19T18:36:39.729 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 2/6 2023-12-19T18:36:39.729 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-19T18:36:39.729 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-19T18:36:39.729 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-16.el9.x86_64 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:40.078 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:36:40.360 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2023-12-19T18:36:40.944 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:41 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:36:40.991 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 5/6 2023-12-19T18:36:41.064 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 6/6 2023-12-19T18:36:41.101 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 6.5 k 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.1 M 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 1.4 M 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 238 k 2023-12-19T18:36:41.104 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 3.7 M 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 15 M 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-19T18:36:41.105 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-19T18:36:41.106 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-19T18:36:41.107 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-19T18:36:41.107 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:41.107 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:36:41.107 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2023-12-19T18:36:41.107 INFO:teuthology.orchestra.run.smithi059.stdout:Install 33 Packages 2023-12-19T18:36:41.107 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:41.107 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 27 M 2023-12-19T18:36:41.107 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 99 M 2023-12-19T18:36:41.108 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:36:41.533 INFO:teuthology.orchestra.run.smithi059.stdout:(1/33): ceph-17.2.7-131.ga88f3edf.el9.x86_64.rp 25 kB/s | 6.5 kB 00:00 2023-12-19T18:36:41.767 INFO:teuthology.orchestra.run.smithi059.stdout:(2/33): ceph-mgr-17.2.7-131.ga88f3edf.el9.x86_6 2.8 MB/s | 1.4 MB 00:00 2023-12-19T18:36:41.817 INFO:teuthology.orchestra.run.smithi059.stdout:(3/33): ceph-mds-17.2.7-131.ga88f3edf.el9.x86_6 3.8 MB/s | 2.1 MB 00:00 2023-12-19T18:36:41.957 INFO:teuthology.orchestra.run.smithi059.stdout:(4/33): ceph-mgr-modules-core-17.2.7-131.ga88f3 1.7 MB/s | 238 kB 00:00 2023-12-19T18:36:42.066 INFO:teuthology.orchestra.run.smithi059.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2023-12-19T18:36:42.099 INFO:teuthology.orchestra.run.smithi059.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2023-12-19T18:36:42.133 INFO:teuthology.orchestra.run.smithi059.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-19T18:36:42.191 INFO:teuthology.orchestra.run.smithi059.stdout:(8/33): ceph-mon-17.2.7-131.ga88f3edf.el9.x86_6 5.6 MB/s | 3.7 MB 00:00 2023-12-19T18:36:42.208 INFO:teuthology.orchestra.run.smithi059.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 513 kB/s | 39 kB 00:00 2023-12-19T18:36:42.242 INFO:teuthology.orchestra.run.smithi059.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-19T18:36:42.300 INFO:teuthology.orchestra.run.smithi059.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2023-12-19T18:36:42.326 INFO:teuthology.orchestra.run.smithi059.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2023-12-19T18:36:42.351 INFO:teuthology.orchestra.run.smithi059.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-19T18:36:42.376 INFO:teuthology.orchestra.run.smithi059.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-19T18:36:42.401 INFO:teuthology.orchestra.run.smithi059.stdout:(15/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2023-12-19T18:36:42.485 INFO:teuthology.orchestra.run.smithi059.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 402 kB/s | 43 kB 00:00 2023-12-19T18:36:42.535 INFO:teuthology.orchestra.run.smithi059.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2023-12-19T18:36:42.686 INFO:teuthology.orchestra.run.smithi059.stdout:(18/33): ceph-osd-17.2.7-131.ga88f3edf.el9.x86_ 17 MB/s | 15 MB 00:00 2023-12-19T18:36:42.711 INFO:teuthology.orchestra.run.smithi059.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 61 kB/s | 11 kB 00:00 2023-12-19T18:36:42.737 INFO:teuthology.orchestra.run.smithi059.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2023-12-19T18:36:42.762 INFO:teuthology.orchestra.run.smithi059.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2023-12-19T18:36:42.787 INFO:teuthology.orchestra.run.smithi059.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2023-12-19T18:36:42.813 INFO:teuthology.orchestra.run.smithi059.stdout:(23/33): python3-jaraco-collections-3.0.0-8.el9 229 kB/s | 23 kB 00:00 2023-12-19T18:36:42.837 INFO:teuthology.orchestra.run.smithi059.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2023-12-19T18:36:42.863 INFO:teuthology.orchestra.run.smithi059.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2023-12-19T18:36:42.888 INFO:teuthology.orchestra.run.smithi059.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-12-19T18:36:42.913 INFO:teuthology.orchestra.run.smithi059.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2023-12-19T18:36:42.939 INFO:teuthology.orchestra.run.smithi059.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2023-12-19T18:36:42.964 INFO:teuthology.orchestra.run.smithi059.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2023-12-19T18:36:42.989 INFO:teuthology.orchestra.run.smithi059.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2023-12-19T18:36:43.014 INFO:teuthology.orchestra.run.smithi059.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2023-12-19T18:36:43.048 INFO:teuthology.orchestra.run.smithi059.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2023-12-19T18:36:43.073 INFO:teuthology.orchestra.run.smithi059.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2023-12-19T18:36:43.076 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:36:43.077 INFO:teuthology.orchestra.run.smithi059.stdout:Total 14 MB/s | 27 MB 00:01 2023-12-19T18:36:43.077 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:36:43.120 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:36:43.120 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:36:43.389 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 6/6 2023-12-19T18:36:43.389 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 1/6 2023-12-19T18:36:43.390 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 2/6 2023-12-19T18:36:43.390 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-19T18:36:43.390 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-19T18:36:43.390 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-19T18:36:43.475 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:36:43.475 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout: jq-1.6-16.el9.x86_64 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:43.779 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:36:44.097 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph 2023-12-19T18:36:44.657 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:39 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:36:44.719 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:36:44.792 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-12-19T18:36:44.804 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:36:44.806 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2023-12-19T18:36:44.806 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-12-19T18:36:44.806 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2023-12-19T18:36:44.806 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:36:44.806 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 6.5 k 2023-12-19T18:36:44.806 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-12-19T18:36:44.806 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.1 M 2023-12-19T18:36:44.806 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 1.4 M 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 238 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 3.7 M 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 15 M 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-19T18:36:44.807 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-19T18:36:44.808 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-19T18:36:44.809 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-19T18:36:44.809 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:44.809 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:36:44.809 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2023-12-19T18:36:44.809 INFO:teuthology.orchestra.run.smithi089.stdout:Install 33 Packages 2023-12-19T18:36:44.809 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:44.809 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 27 M 2023-12-19T18:36:44.810 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 99 M 2023-12-19T18:36:44.810 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:36:44.834 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-12-19T18:36:44.866 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-12-19T18:36:44.961 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-12-19T18:36:45.046 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-12-19T18:36:45.094 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-12-19T18:36:45.099 INFO:teuthology.orchestra.run.smithi089.stdout:(1/33): ceph-17.2.7-131.ga88f3edf.el9.x86_64.rp 27 kB/s | 6.5 kB 00:00 2023-12-19T18:36:45.167 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-12-19T18:36:45.220 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-12-19T18:36:45.241 INFO:teuthology.orchestra.run.smithi089.stdout:(2/33): ceph-mds-17.2.7-131.ga88f3edf.el9.x86_6 5.4 MB/s | 2.1 MB 00:00 2023-12-19T18:36:45.258 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-12-19T18:36:45.316 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-12-19T18:36:45.371 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-12-19T18:36:45.562 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-12-19T18:36:45.620 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-12-19T18:36:45.671 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-12-19T18:36:45.721 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-12-19T18:36:45.748 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-12-19T18:36:45.827 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-12-19T18:36:45.858 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-19T18:36:45.883 INFO:teuthology.orchestra.run.smithi089.stdout:(3/33): ceph-mgr-17.2.7-131.ga88f3edf.el9.x86_6 1.4 MB/s | 1.4 MB 00:01 2023-12-19T18:36:45.892 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-12-19T18:36:45.924 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-12-19T18:36:45.968 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-12-19T18:36:46.002 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-12-19T18:36:46.066 INFO:teuthology.orchestra.run.smithi089.stdout:(4/33): ceph-osd-17.2.7-131.ga88f3edf.el9.x86_6 19 MB/s | 15 MB 00:00 2023-12-19T18:36:46.092 INFO:teuthology.orchestra.run.smithi089.stdout:(5/33): ceph-mgr-modules-core-17.2.7-131.ga88f3 1.1 MB/s | 238 kB 00:00 2023-12-19T18:36:46.094 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-12-19T18:36:46.135 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-12-19T18:36:46.184 INFO:teuthology.orchestra.run.smithi089.stdout:(6/33): ceph-mon-17.2.7-131.ga88f3edf.el9.x86_6 3.4 MB/s | 3.7 MB 00:01 2023-12-19T18:36:46.215 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-12-19T18:36:46.284 INFO:teuthology.orchestra.run.smithi089.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 664 kB/s | 144 kB 00:00 2023-12-19T18:36:46.301 INFO:teuthology.orchestra.run.smithi089.stdout:(8/33): python3-markupsafe-1.1.1-12.el9.x86_64. 171 kB/s | 32 kB 00:00 2023-12-19T18:36:46.318 INFO:teuthology.orchestra.run.smithi089.stdout:(9/33): python3-pytz-2021.1-5.el9.noarch.rpm 435 kB/s | 47 kB 00:00 2023-12-19T18:36:46.335 INFO:teuthology.orchestra.run.smithi089.stdout:(10/33): python3-toml-0.10.2-6.el9.noarch.rpm 761 kB/s | 39 kB 00:00 2023-12-19T18:36:46.358 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-12-19T18:36:46.478 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-19T18:36:46.510 INFO:teuthology.orchestra.run.smithi089.stdout:(11/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 241 kB 00:00 2023-12-19T18:36:46.602 INFO:teuthology.orchestra.run.smithi089.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 4.0 MB/s | 1.1 MB 00:00 2023-12-19T18:36:46.636 INFO:teuthology.orchestra.run.smithi089.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 342 kB/s | 103 kB 00:00 2023-12-19T18:36:46.637 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.n 28/33 2023-12-19T18:36:46.661 INFO:teuthology.orchestra.run.smithi089.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 824 kB/s | 124 kB 00:00 2023-12-19T18:36:46.662 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 29/33 2023-12-19T18:36:46.687 INFO:teuthology.orchestra.run.smithi089.stdout:(15/33): python3-requests-2.25.1-7.el9.noarch.r 1.3 MB/s | 113 kB 00:00 2023-12-19T18:36:46.689 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 29/33 2023-12-19T18:36:46.689 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:46.689 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-19T18:36:46.689 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T18:36:46.689 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T18:36:46.689 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:46.721 INFO:teuthology.orchestra.run.smithi089.stdout:(16/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2023-12-19T18:36:46.829 INFO:teuthology.orchestra.run.smithi089.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2023-12-19T18:36:46.871 INFO:teuthology.orchestra.run.smithi089.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 262 kB/s | 11 kB 00:00 2023-12-19T18:36:46.896 INFO:teuthology.orchestra.run.smithi089.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 822 kB/s | 172 kB 00:00 2023-12-19T18:36:46.922 INFO:teuthology.orchestra.run.smithi089.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2023-12-19T18:36:46.947 INFO:teuthology.orchestra.run.smithi089.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2023-12-19T18:36:46.973 INFO:teuthology.orchestra.run.smithi089.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2023-12-19T18:36:46.997 INFO:teuthology.orchestra.run.smithi089.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2023-12-19T18:36:47.023 INFO:teuthology.orchestra.run.smithi089.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2023-12-19T18:36:47.048 INFO:teuthology.orchestra.run.smithi089.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2023-12-19T18:36:47.073 INFO:teuthology.orchestra.run.smithi089.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-12-19T18:36:47.107 INFO:teuthology.orchestra.run.smithi089.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2023-12-19T18:36:47.132 INFO:teuthology.orchestra.run.smithi089.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 283 kB/s | 16 kB 00:00 2023-12-19T18:36:47.158 INFO:teuthology.orchestra.run.smithi089.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2023-12-19T18:36:47.183 INFO:teuthology.orchestra.run.smithi089.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2023-12-19T18:36:47.208 INFO:teuthology.orchestra.run.smithi089.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2023-12-19T18:36:47.250 INFO:teuthology.orchestra.run.smithi089.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2023-12-19T18:36:47.275 INFO:teuthology.orchestra.run.smithi089.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2023-12-19T18:36:47.280 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:36:47.281 INFO:teuthology.orchestra.run.smithi089.stdout:Total 11 MB/s | 27 MB 00:02 2023-12-19T18:36:47.281 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:36:47.315 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:36:47.315 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:36:47.652 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:36:47.652 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:36:48.104 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 30/33 2023-12-19T18:36:48.132 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 30/33 2023-12-19T18:36:48.132 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:48.132 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-19T18:36:48.132 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T18:36:48.132 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T18:36:48.133 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:48.530 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 31/33 2023-12-19T18:36:48.555 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 31/33 2023-12-19T18:36:48.555 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:48.555 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-19T18:36:48.555 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T18:36:48.555 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T18:36:48.555 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:48.772 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 32/33 2023-12-19T18:36:48.797 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 32/33 2023-12-19T18:36:48.797 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:48.797 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-19T18:36:48.797 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T18:36:48.797 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T18:36:48.797 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:48.864 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 33/33 2023-12-19T18:36:48.869 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:36:48.953 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-12-19T18:36:48.985 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-12-19T18:36:49.027 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-12-19T18:36:49.118 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-12-19T18:36:49.188 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-12-19T18:36:49.230 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-12-19T18:36:49.301 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-12-19T18:36:49.363 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-12-19T18:36:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-12-19T18:36:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-12-19T18:36:49.507 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-12-19T18:36:49.690 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-12-19T18:36:49.740 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-12-19T18:36:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-12-19T18:36:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-12-19T18:36:49.860 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-12-19T18:36:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-12-19T18:36:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-19T18:36:50.005 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-12-19T18:36:50.045 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-12-19T18:36:50.080 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-12-19T18:36:50.115 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-12-19T18:36:50.203 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-12-19T18:36:50.243 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-12-19T18:36:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-12-19T18:36:50.470 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-12-19T18:36:50.542 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 33/33 2023-12-19T18:36:50.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 1/33 2023-12-19T18:36:50.542 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 2/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 3/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 4/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 5/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.n 6/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-12-19T18:36:50.543 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-19T18:36:50.544 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-12-19T18:36:50.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-12-19T18:36:50.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-12-19T18:36:50.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-12-19T18:36:50.545 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-12-19T18:36:50.587 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-19T18:36:50.737 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.n 28/33 2023-12-19T18:36:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 29/33 2023-12-19T18:36:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 29/33 2023-12-19T18:36:50.793 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:50.793 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-19T18:36:50.793 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T18:36:50.793 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T18:36:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:51.504 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-12-19T18:36:51.504 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:51.504 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:36:51.504 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:51.504 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:51.505 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:51.505 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:36:51.505 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:51.505 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:51.505 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-19T18:36:51.505 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-19T18:36:51.505 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-19T18:36:51.506 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-19T18:36:51.506 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-19T18:36:51.506 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-19T18:36:51.506 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-19T18:36:51.506 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-19T18:36:51.506 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-19T18:36:51.506 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-19T18:36:51.507 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-19T18:36:51.507 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-19T18:36:51.507 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-19T18:36:51.507 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-19T18:36:51.507 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-19T18:36:51.507 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ply-3.11-14.el9.noarch 2023-12-19T18:36:51.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-19T18:36:51.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-19T18:36:51.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-19T18:36:51.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-19T18:36:51.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-19T18:36:51.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-19T18:36:51.508 INFO:teuthology.orchestra.run.smithi059.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-19T18:36:51.509 INFO:teuthology.orchestra.run.smithi059.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-19T18:36:51.509 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-19T18:36:51.509 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-19T18:36:51.509 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-19T18:36:51.509 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:51.509 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:36:51.887 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2023-12-19T18:36:52.186 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 30/33 2023-12-19T18:36:52.215 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 30/33 2023-12-19T18:36:52.215 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:52.215 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-19T18:36:52.215 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T18:36:52.215 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T18:36:52.215 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:52.467 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:53 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:36:52.563 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:36:52.587 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 31/33 2023-12-19T18:36:52.611 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 31/33 2023-12-19T18:36:52.611 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:52.611 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-19T18:36:52.611 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T18:36:52.611 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T18:36:52.611 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:52.614 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:36:52.616 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-19T18:36:52.616 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:36:52.673 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2023-12-19T18:36:52.813 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 32/33 2023-12-19T18:36:52.836 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 32/33 2023-12-19T18:36:52.836 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:52.836 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-19T18:36:52.836 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T18:36:52.836 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T18:36:52.836 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:52.896 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 33/33 2023-12-19T18:36:53.254 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:54 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:36:53.402 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:36:53.403 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:53.403 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-19T18:36:53.403 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:53.403 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:36:53.403 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 80 k 2023-12-19T18:36:53.404 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:53.404 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:36:53.404 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:53.404 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-19T18:36:53.404 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:53.404 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 80 k 2023-12-19T18:36:53.404 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 364 k 2023-12-19T18:36:53.404 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:36:54.070 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-17.2.7-131.ga88f3edf.el9.noarch.rpm 121 kB/s | 80 kB 00:00 2023-12-19T18:36:54.071 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:36:54.072 INFO:teuthology.orchestra.run.smithi059.stdout:Total 120 kB/s | 80 kB 00:00 2023-12-19T18:36:54.072 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:36:54.077 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:36:54.077 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:36:54.079 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:36:54.079 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:36:54.163 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:36:54.583 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 33/33 2023-12-19T18:36:54.583 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 1/33 2023-12-19T18:36:54.584 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 2/33 2023-12-19T18:36:54.584 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 3/33 2023-12-19T18:36:54.584 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 4/33 2023-12-19T18:36:54.584 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 5/33 2023-12-19T18:36:54.584 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.n 6/33 2023-12-19T18:36:54.584 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-12-19T18:36:54.584 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-12-19T18:36:54.584 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-12-19T18:36:54.585 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-12-19T18:36:54.585 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-12-19T18:36:54.585 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-12-19T18:36:54.585 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-12-19T18:36:54.585 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-12-19T18:36:54.585 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-12-19T18:36:54.585 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-12-19T18:36:54.586 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-12-19T18:36:54.586 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-19T18:36:54.586 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-12-19T18:36:54.586 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-12-19T18:36:54.586 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-12-19T18:36:54.586 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-12-19T18:36:54.586 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-12-19T18:36:54.586 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-12-19T18:36:54.586 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-12-19T18:36:54.588 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-12-19T18:36:54.588 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-19T18:36:54.588 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-12-19T18:36:54.588 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-12-19T18:36:54.588 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-12-19T18:36:54.588 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-12-19T18:36:54.588 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-12-19T18:36:54.628 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T18:36:54.727 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T18:36:55.105 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T18:36:55.417 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T18:36:55.418 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:55.418 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:36:55.418 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:36:55.418 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:55.418 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:36:55.516 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-19T18:36:55.517 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-19T18:36:55.518 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-19T18:36:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-19T18:36:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-19T18:36:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-19T18:36:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-19T18:36:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-19T18:36:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-19T18:36:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-19T18:36:55.519 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:55.519 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:36:55.633 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2023-12-19T18:36:55.894 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-base 2023-12-19T18:36:56.214 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:57 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:36:56.366 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:36:56.366 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-12-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 137 k 2023-12-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:56.368 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 137 k 2023-12-19T18:36:56.368 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 378 k 2023-12-19T18:36:56.368 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:36:56.452 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:51 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:36:56.545 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:36:56.593 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:36:56.594 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-12-19T18:36:56.594 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:36:56.673 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install cephadm 2023-12-19T18:36:56.984 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-17.2.7-131.ga88f3ed 222 kB/s | 137 kB 00:00 2023-12-19T18:36:56.985 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:36:56.985 INFO:teuthology.orchestra.run.smithi059.stdout:Total 222 kB/s | 137 kB 00:00 2023-12-19T18:36:56.985 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:36:56.997 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:36:56.997 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:36:57.049 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:36:57.049 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:36:57.190 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:36:57.223 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:52 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:36:57.252 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T18:36:57.276 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T18:36:57.277 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:36:57.277 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-19T18:36:57.277 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:57.366 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 80 k 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2023-12-19T18:36:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:57.368 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 80 k 2023-12-19T18:36:57.368 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 364 k 2023-12-19T18:36:57.368 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:36:57.872 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm-17.2.7-131.ga88f3edf.el9.noarch.rpm 160 kB/s | 80 kB 00:00 2023-12-19T18:36:57.873 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:36:57.873 INFO:teuthology.orchestra.run.smithi089.stdout:Total 159 kB/s | 80 kB 00:00 2023-12-19T18:36:57.873 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:36:57.878 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:36:57.878 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:36:57.880 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:36:57.880 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:36:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:36:58.307 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T18:36:58.308 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:58.308 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:36:58.308 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:36:58.308 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:36:58.308 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:36:58.444 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T18:36:58.521 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2023-12-19T18:36:58.535 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T18:36:58.915 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T18:36:59.102 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:00 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:36:59.199 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:36:59.206 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T18:36:59.206 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:59.206 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:36:59.206 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:36:59.206 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:36:59.207 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:36:59.251 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:36:59.252 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-19T18:36:59.252 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:36:59.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2023-12-19T18:36:59.414 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-immutable-object-cache 2023-12-19T18:36:59.897 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:00 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:36:59.969 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:54 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:00.048 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:00.049 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:00.049 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-19T18:37:00.049 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:00.049 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:00.049 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 1.7 M 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 23 k 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 15 k 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout:Install 7 Packages 2023-12-19T18:37:00.050 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:00.051 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.0 M 2023-12-19T18:37:00.051 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 8.1 M 2023-12-19T18:37:00.051 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:00.119 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:00.119 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:00.119 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2023-12-19T18:37:00.119 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 137 k 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 137 k 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 378 k 2023-12-19T18:37:00.120 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:00.562 INFO:teuthology.orchestra.run.smithi059.stdout:(1/7): ceph-grafana-dashboards-17.2.7-131.ga88f 55 kB/s | 23 kB 00:00 2023-12-19T18:37:00.754 INFO:teuthology.orchestra.run.smithi059.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 47 kB/s | 9.0 kB 00:00 2023-12-19T18:37:00.813 INFO:teuthology.orchestra.run.smithi059.stdout:(3/7): ceph-prometheus-alerts-17.2.7-131.ga88f3 22 kB/s | 15 kB 00:00 2023-12-19T18:37:00.938 INFO:teuthology.orchestra.run.smithi059.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 223 kB/s | 41 kB 00:00 2023-12-19T18:37:01.138 INFO:teuthology.orchestra.run.smithi059.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 94 kB/s | 31 kB 00:00 2023-12-19T18:37:01.197 INFO:teuthology.orchestra.run.smithi059.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 729 kB/s | 188 kB 00:00 2023-12-19T18:37:01.237 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-immutable-object-cache-17.2.7-131.ga88f3ed 123 kB/s | 137 kB 00:01 2023-12-19T18:37:01.238 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:01.239 INFO:teuthology.orchestra.run.smithi089.stdout:Total 122 kB/s | 137 kB 00:01 2023-12-19T18:37:01.239 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:01.248 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:01.248 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:01.303 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:01.304 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:01.446 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:01.538 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T18:37:01.563 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T18:37:01.564 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:37:01.564 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-19T18:37:01.564 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:01.564 INFO:teuthology.orchestra.run.smithi059.stdout:(7/7): ceph-mgr-dashboard-17.2.7-131.ga88f3edf. 1.2 MB/s | 1.7 MB 00:01 2023-12-19T18:37:01.565 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:01.566 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2023-12-19T18:37:01.566 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:01.576 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:01.576 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:01.659 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:01.662 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:01.969 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:02.031 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-19T18:37:02.066 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-19T18:37:02.112 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-19T18:37:02.143 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-19T18:37:02.172 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.no 5/7 2023-12-19T18:37:02.412 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.n 6/7 2023-12-19T18:37:02.442 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 7/7 2023-12-19T18:37:02.569 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T18:37:02.569 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:02.569 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:37:02.569 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:37:02.569 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:02.570 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:02.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr 2023-12-19T18:37:02.996 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 7/7 2023-12-19T18:37:02.997 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.n 1/7 2023-12-19T18:37:02.997 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 2/7 2023-12-19T18:37:02.997 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.no 3/7 2023-12-19T18:37:02.997 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-19T18:37:02.997 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-19T18:37:02.997 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-19T18:37:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-19T18:37:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:03.334 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:03.334 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-19T18:37:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-19T18:37:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-19T18:37:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-19T18:37:03.335 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:03.335 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:03.366 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:58 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:03.459 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:03.508 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:03.509 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-12-19T18:37:03.509 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:03.545 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-19T18:37:03.584 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-dashboard 2023-12-19T18:37:04.131 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:59 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:04.145 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:05 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:04.276 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:04.276 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 1.7 M 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 23 k 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 15 k 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout:Installing weak dependencies: 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:04.277 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:04.278 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:04.278 INFO:teuthology.orchestra.run.smithi089.stdout:Install 7 Packages 2023-12-19T18:37:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:04.278 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.0 M 2023-12-19T18:37:04.278 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 8.1 M 2023-12-19T18:37:04.278 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:04.309 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:04.310 INFO:teuthology.orchestra.run.smithi059.stdout:=========================================================================================== 2023-12-19T18:37:04.310 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-19T18:37:04.310 INFO:teuthology.orchestra.run.smithi059.stdout:=========================================================================================== 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 7.4 M 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-19T18:37:04.311 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-19T18:37:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-19T18:37:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-19T18:37:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-19T18:37:04.312 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:04.312 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:04.312 INFO:teuthology.orchestra.run.smithi059.stdout:=========================================================================================== 2023-12-19T18:37:04.313 INFO:teuthology.orchestra.run.smithi059.stdout:Install 12 Packages 2023-12-19T18:37:04.313 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:04.314 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 37 M 2023-12-19T18:37:04.314 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 209 M 2023-12-19T18:37:04.314 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:04.704 INFO:teuthology.orchestra.run.smithi059.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 268 kB/s | 32 kB 00:00 2023-12-19T18:37:04.729 INFO:teuthology.orchestra.run.smithi059.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 688 kB/s | 17 kB 00:00 2023-12-19T18:37:04.751 INFO:teuthology.orchestra.run.smithi089.stdout:(1/7): ceph-grafana-dashboards-17.2.7-131.ga88f 59 kB/s | 23 kB 00:00 2023-12-19T18:37:04.762 INFO:teuthology.orchestra.run.smithi059.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-12-19T18:37:04.793 INFO:teuthology.orchestra.run.smithi089.stdout:(2/7): ceph-prometheus-alerts-17.2.7-131.ga88f3 34 kB/s | 15 kB 00:00 2023-12-19T18:37:04.838 INFO:teuthology.orchestra.run.smithi059.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-12-19T18:37:05.038 INFO:teuthology.orchestra.run.smithi059.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-12-19T18:37:05.076 INFO:teuthology.orchestra.run.smithi089.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 28 kB/s | 9.0 kB 00:00 2023-12-19T18:37:05.105 INFO:teuthology.orchestra.run.smithi059.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2023-12-19T18:37:05.130 INFO:teuthology.orchestra.run.smithi059.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-19T18:37:05.268 INFO:teuthology.orchestra.run.smithi089.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 86 kB/s | 41 kB 00:00 2023-12-19T18:37:05.352 INFO:teuthology.orchestra.run.smithi089.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 112 kB/s | 31 kB 00:00 2023-12-19T18:37:05.639 INFO:teuthology.orchestra.run.smithi059.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2023-12-19T18:37:05.664 INFO:teuthology.orchestra.run.smithi059.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2023-12-19T18:37:05.689 INFO:teuthology.orchestra.run.smithi059.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2023-12-19T18:37:05.735 INFO:teuthology.orchestra.run.smithi089.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 404 kB/s | 188 kB 00:00 2023-12-19T18:37:06.431 INFO:teuthology.orchestra.run.smithi059.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 320 kB/s | 245 kB 00:00 2023-12-19T18:37:07.094 INFO:teuthology.orchestra.run.smithi089.stdout:(7/7): ceph-mgr-dashboard-17.2.7-131.ga88f3edf. 635 kB/s | 1.7 MB 00:02 2023-12-19T18:37:07.097 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:07.098 INFO:teuthology.orchestra.run.smithi089.stdout:Total 724 kB/s | 2.0 MB 00:02 2023-12-19T18:37:07.098 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:07.107 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:07.107 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:07.178 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:07.178 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:07.476 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:07.506 INFO:teuthology.orchestra.run.smithi059.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-1 2.5 MB/s | 7.4 MB 00:02 2023-12-19T18:37:07.510 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:07.510 INFO:teuthology.orchestra.run.smithi059.stdout:Total 12 MB/s | 37 MB 00:03 2023-12-19T18:37:07.525 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:07.537 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-19T18:37:07.553 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:07.554 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:07.571 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-19T18:37:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-19T18:37:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-19T18:37:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.no 5/7 2023-12-19T18:37:07.907 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.n 6/7 2023-12-19T18:37:07.940 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 7/7 2023-12-19T18:37:07.995 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:07.995 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 7/7 2023-12-19T18:37:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.n 1/7 2023-12-19T18:37:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 2/7 2023-12-19T18:37:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.no 3/7 2023-12-19T18:37:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-19T18:37:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-19T18:37:08.479 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-19T18:37:08.785 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-19T18:37:08.785 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:08.786 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:08.816 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:08.888 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-19T18:37:08.919 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-19T18:37:09.021 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-19T18:37:09.031 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-19T18:37:09.055 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-19T18:37:09.241 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T18:37:09.271 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T18:37:09.354 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-19T18:37:09.586 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:04 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:09.733 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:09.734 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================== 2023-12-19T18:37:09.734 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-12-19T18:37:09.734 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================== 2023-12-19T18:37:09.734 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:09.734 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 7.4 M 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:09.735 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:09.736 INFO:teuthology.orchestra.run.smithi089.stdout:=========================================================================================== 2023-12-19T18:37:09.736 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2023-12-19T18:37:09.736 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:09.736 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 37 M 2023-12-19T18:37:09.736 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 209 M 2023-12-19T18:37:09.736 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:09.918 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 270 kB/s | 32 kB 00:00 2023-12-19T18:37:09.969 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 647 kB/s | 17 kB 00:00 2023-12-19T18:37:10.002 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-12-19T18:37:10.102 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-19T18:37:10.178 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2023-12-19T18:37:10.216 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-19T18:37:10.320 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2023-12-19T18:37:10.362 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2023-12-19T18:37:10.429 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2023-12-19T18:37:10.779 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2023-12-19T18:37:10.805 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2023-12-19T18:37:10.830 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.9 MB/s | 195 kB 00:00 2023-12-19T18:37:11.022 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2023-12-19T18:37:11.574 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-19T18:37:12.647 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-19T18:37:12.669 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 12/12 2023-12-19T18:37:14.181 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-1 1.7 MB/s | 7.4 MB 00:04 2023-12-19T18:37:14.183 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:14.183 INFO:teuthology.orchestra.run.smithi089.stdout:Total 8.3 MB/s | 37 MB 00:04 2023-12-19T18:37:14.198 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:14.224 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:14.224 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:14.652 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:14.701 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:15.096 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 12/12 2023-12-19T18:37:15.096 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 1/12 2023-12-19T18:37:15.096 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-19T18:37:15.096 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-19T18:37:15.096 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-19T18:37:15.096 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T18:37:15.096 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T18:37:15.096 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-19T18:37:15.097 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-19T18:37:15.097 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-19T18:37:15.097 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-19T18:37:15.097 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-19T18:37:15.350 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:15.419 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-19T18:37:15.459 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-19T18:37:15.552 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-19T18:37:15.552 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:15.552 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:15.552 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:15.552 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-19T18:37:15.552 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-19T18:37:15.552 INFO:teuthology.orchestra.run.smithi059.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-19T18:37:15.552 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi059.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:15.553 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-19T18:37:15.596 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-19T18:37:15.774 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T18:37:15.803 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T18:37:15.882 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-19T18:37:15.929 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2023-12-19T18:37:16.516 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:17 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:16.539 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-19T18:37:16.642 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-19T18:37:16.669 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:16.670 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2023-12-19T18:37:16.670 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-19T18:37:16.670 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2023-12-19T18:37:16.670 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:16.670 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 47 k 2023-12-19T18:37:16.670 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-19T18:37:16.670 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-19T18:37:16.670 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2023-12-19T18:37:16.671 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2023-12-19T18:37:16.672 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:16.672 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.0 M 2023-12-19T18:37:16.672 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 26 M 2023-12-19T18:37:16.672 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:16.935 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 245 kB/s | 24 kB 00:00 2023-12-19T18:37:16.986 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2023-12-19T18:37:17.044 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2023-12-19T18:37:17.069 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2023-12-19T18:37:17.103 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 211 kB 00:00 2023-12-19T18:37:17.128 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 734 kB/s | 43 kB 00:00 2023-12-19T18:37:17.478 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): ceph-mgr-rook-17.2.7-131.ga88f3edf.el9. 74 kB/s | 47 kB 00:00 2023-12-19T18:37:17.504 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 80 kB/s | 32 kB 00:00 2023-12-19T18:37:17.587 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 33 kB/s | 15 kB 00:00 2023-12-19T18:37:17.883 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-19T18:37:17.988 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 148 kB/s | 59 kB 00:00 2023-12-19T18:37:18.054 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2023-12-19T18:37:18.088 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-google-auth-2.23.4-1.el9.noarc 324 kB/s | 197 kB 00:00 2023-12-19T18:37:18.238 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 359 kB/s | 90 kB 00:00 2023-12-19T18:37:18.240 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:18.240 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2023-12-19T18:37:18.241 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:18.257 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:18.257 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:18.371 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:18.372 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:18.910 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:18.932 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-19T18:37:18.972 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 12/12 2023-12-19T18:37:19.084 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-19T18:37:19.140 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-19T18:37:19.204 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-19T18:37:19.240 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-19T18:37:19.282 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-19T18:37:19.378 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-19T18:37:19.488 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-19T18:37:19.543 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-19T18:37:20.306 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-19T18:37:20.355 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-19T18:37:20.389 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-19T18:37:20.435 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-19T18:37:20.466 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 13/13 2023-12-19T18:37:21.227 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 13/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 1/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-19T18:37:21.228 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-19T18:37:21.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-19T18:37:21.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-19T18:37:21.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-19T18:37:21.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-19T18:37:21.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-19T18:37:21.608 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:21.608 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 12/12 2023-12-19T18:37:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 1/12 2023-12-19T18:37:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-19T18:37:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-19T18:37:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-19T18:37:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T18:37:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T18:37:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-19T18:37:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-19T18:37:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-19T18:37:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-19T18:37:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-19T18:37:21.841 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2023-12-19T18:37:22.193 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-19T18:37:22.193 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:22.193 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:22.194 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:22.425 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:23 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:22.577 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-rook 2023-12-19T18:37:22.578 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 121 k 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-19T18:37:22.579 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:22.580 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:22.580 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================= 2023-12-19T18:37:22.580 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2023-12-19T18:37:22.580 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:22.581 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 6.8 M 2023-12-19T18:37:22.581 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 33 M 2023-12-19T18:37:22.581 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:22.916 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-19T18:37:23.099 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-12-19T18:37:23.159 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:18 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:23.312 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 47 k 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-19T18:37:23.314 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout:Install 13 Packages 2023-12-19T18:37:23.315 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:23.316 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.0 M 2023-12-19T18:37:23.316 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 26 M 2023-12-19T18:37:23.316 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:23.499 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): ceph-mgr-cephadm-17.2.7-131.ga88f3edf.el 163 kB/s | 121 kB 00:00 2023-12-19T18:37:23.516 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 138 kB/s | 58 kB 00:00 2023-12-19T18:37:23.607 INFO:teuthology.orchestra.run.smithi089.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 193 kB/s | 24 kB 00:00 2023-12-19T18:37:23.608 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 792 kB/s | 548 kB 00:00 2023-12-19T18:37:23.658 INFO:teuthology.orchestra.run.smithi089.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 102 kB/s | 18 kB 00:00 2023-12-19T18:37:23.700 INFO:teuthology.orchestra.run.smithi089.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2023-12-19T18:37:23.725 INFO:teuthology.orchestra.run.smithi089.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-19T18:37:23.750 INFO:teuthology.orchestra.run.smithi089.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-19T18:37:23.776 INFO:teuthology.orchestra.run.smithi089.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2023-12-19T18:37:23.917 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 122 kB/s | 51 kB 00:00 2023-12-19T18:37:23.919 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:23.919 INFO:teuthology.orchestra.run.smithi059.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2023-12-19T18:37:23.919 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:23.930 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:23.930 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:23.951 INFO:teuthology.orchestra.run.smithi089.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 161 kB/s | 32 kB 00:00 2023-12-19T18:37:24.029 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:24.030 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:24.152 INFO:teuthology.orchestra.run.smithi089.stdout:(8/13): ceph-mgr-rook-17.2.7-131.ga88f3edf.el9. 71 kB/s | 47 kB 00:00 2023-12-19T18:37:24.177 INFO:teuthology.orchestra.run.smithi089.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2023-12-19T18:37:24.196 INFO:teuthology.orchestra.run.smithi089.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 808 kB/s | 197 kB 00:00 2023-12-19T18:37:24.360 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:24.396 INFO:teuthology.orchestra.run.smithi089.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 4.2 MB/s | 1.0 MB 00:00 2023-12-19T18:37:24.413 INFO:teuthology.orchestra.run.smithi089.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 272 kB/s | 59 kB 00:00 2023-12-19T18:37:24.454 INFO:teuthology.orchestra.run.smithi089.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 347 kB/s | 90 kB 00:00 2023-12-19T18:37:24.458 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:24.458 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2023-12-19T18:37:24.459 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:24.471 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:24.471 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:24.541 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-19T18:37:24.566 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:24.568 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:24.606 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-19T18:37:25.080 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:25.167 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-19T18:37:25.242 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-19T18:37:25.249 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-19T18:37:25.304 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-19T18:37:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-19T18:37:25.331 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 6/6 2023-12-19T18:37:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-19T18:37:25.391 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-19T18:37:25.431 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-19T18:37:25.533 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-19T18:37:25.643 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-19T18:37:25.692 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-19T18:37:26.098 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 6/6 2023-12-19T18:37:26.098 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/6 2023-12-19T18:37:26.098 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-19T18:37:26.098 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-19T18:37:26.098 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-19T18:37:26.098 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-19T18:37:26.415 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-19T18:37:26.415 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-19T18:37:26.415 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:26.415 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:26.416 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:26.416 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-19T18:37:26.416 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-19T18:37:26.416 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-19T18:37:26.416 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-19T18:37:26.416 INFO:teuthology.orchestra.run.smithi059.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-19T18:37:26.416 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:26.416 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-19T18:37:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-19T18:37:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-19T18:37:26.568 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 13/13 2023-12-19T18:37:26.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2023-12-19T18:37:27.229 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:28 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:27.382 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 810 k 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2023-12-19T18:37:27.383 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:27.384 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 889 k 2023-12-19T18:37:27.384 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 2.6 M 2023-12-19T18:37:27.384 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 13/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 1/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-19T18:37:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-19T18:37:27.502 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 673 kB/s | 79 kB 00:00 2023-12-19T18:37:27.768 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-fuse-17.2.7-131.ga88f3edf.el9.x86_6 2.1 MB/s | 810 kB 00:00 2023-12-19T18:37:27.769 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:27.770 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.2 MB/s | 889 kB 00:00 2023-12-19T18:37:27.770 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:27.792 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:27.792 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-19T18:37:27.810 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-19T18:37:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-19T18:37:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-19T18:37:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-19T18:37:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-19T18:37:27.811 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:27.811 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:27.872 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:27.872 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:28.072 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:28.074 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-cephadm 2023-12-19T18:37:28.201 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-19T18:37:28.278 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2/2 2023-12-19T18:37:28.632 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:23 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:28.777 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:28.778 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2023-12-19T18:37:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-12-19T18:37:28.778 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2023-12-19T18:37:28.778 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 121 k 2023-12-19T18:37:28.778 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-12-19T18:37:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-19T18:37:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-19T18:37:28.778 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-19T18:37:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-19T18:37:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-19T18:37:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:28.779 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:28.779 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================= 2023-12-19T18:37:28.779 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2023-12-19T18:37:28.779 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:28.779 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 6.8 M 2023-12-19T18:37:28.780 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 33 M 2023-12-19T18:37:28.780 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:28.988 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2/2 2023-12-19T18:37:28.988 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/2 2023-12-19T18:37:29.308 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-12-19T18:37:29.313 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-19T18:37:29.313 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:29.313 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:29.313 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-19T18:37:29.313 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:29.313 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:29.458 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2023-12-19T18:37:29.523 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2023-12-19T18:37:29.792 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): ceph-mgr-cephadm-17.2.7-131.ga88f3edf.el 186 kB/s | 121 kB 00:00 2023-12-19T18:37:29.835 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 164 kB/s | 58 kB 00:00 2023-12-19T18:37:29.876 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 968 kB/s | 548 kB 00:00 2023-12-19T18:37:29.918 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 407 kB/s | 51 kB 00:00 2023-12-19T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout:Total 5.9 MB/s | 6.8 MB 00:01 2023-12-19T18:37:29.923 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:29.930 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:29.930 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:30.018 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:30.020 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:30.112 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:31 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:30.264 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 126 k 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:30.265 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 126 k 2023-12-19T18:37:30.266 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 448 k 2023-12-19T18:37:30.266 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:30.320 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:30.485 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-19T18:37:30.553 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-19T18:37:30.788 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-17.2.7-131.ga88f3edf.el9.x86_64. 242 kB/s | 126 kB 00:00 2023-12-19T18:37:30.789 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:30.789 INFO:teuthology.orchestra.run.smithi059.stdout:Total 241 kB/s | 126 kB 00:00 2023-12-19T18:37:30.789 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:30.796 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:30.796 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:30.855 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:30.855 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:31.001 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:31.066 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-19T18:37:31.106 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:31.147 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-19T18:37:31.202 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-19T18:37:31.233 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 6/6 2023-12-19T18:37:31.549 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:31.863 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:31.863 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:31.863 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:31.863 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:37:31.863 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:31.863 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:32.022 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 6/6 2023-12-19T18:37:32.022 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/6 2023-12-19T18:37:32.022 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-19T18:37:32.022 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-19T18:37:32.022 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-19T18:37:32.022 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-19T18:37:32.097 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2023-12-19T18:37:32.340 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-19T18:37:32.340 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:32.340 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:37:32.340 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T18:37:32.340 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-19T18:37:32.340 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-19T18:37:32.341 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-19T18:37:32.341 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-19T18:37:32.341 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-19T18:37:32.341 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:32.341 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:32.599 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-fuse 2023-12-19T18:37:32.685 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:33 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:32.783 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:32.836 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:32.838 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-19T18:37:32.838 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:32.905 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2023-12-19T18:37:33.152 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:28 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:33.296 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:33.297 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:33.297 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2023-12-19T18:37:33.297 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:33.297 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:33.297 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 810 k 2023-12-19T18:37:33.297 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2023-12-19T18:37:33.297 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-19T18:37:33.297 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:33.297 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:33.298 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:33.298 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2023-12-19T18:37:33.298 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:33.298 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 889 k 2023-12-19T18:37:33.298 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 2.6 M 2023-12-19T18:37:33.298 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:33.404 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 745 kB/s | 79 kB 00:00 2023-12-19T18:37:33.490 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:34 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:33.644 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:33.644 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:33.644 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 25 k 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 25 k 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 110 k 2023-12-19T18:37:33.645 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:34.082 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-17.2.7-131.ga88f3edf.el9.x86_64 57 kB/s | 25 kB 00:00 2023-12-19T18:37:34.083 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:34.083 INFO:teuthology.orchestra.run.smithi059.stdout:Total 57 kB/s | 25 kB 00:00 2023-12-19T18:37:34.083 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:34.088 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:34.088 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:34.089 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:34.089 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:34.168 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:34.284 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:34.704 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:34.805 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): ceph-fuse-17.2.7-131.ga88f3edf.el9.x86_6 538 kB/s | 810 kB 00:01 2023-12-19T18:37:34.806 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:34.807 INFO:teuthology.orchestra.run.smithi089.stdout:Total 589 kB/s | 889 kB 00:01 2023-12-19T18:37:34.807 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:34.828 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:34.828 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:34.903 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:34.903 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:34.987 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:34.987 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:34.987 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:34.987 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:37:34.987 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:34.987 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:35.175 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2023-12-19T18:37:35.224 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-19T18:37:35.288 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2/2 2023-12-19T18:37:35.757 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:36 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:35.855 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:35.908 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:35.910 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-19T18:37:35.910 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:35.967 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2023-12-19T18:37:35.976 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2/2 2023-12-19T18:37:35.976 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/2 2023-12-19T18:37:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-19T18:37:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:36.270 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:37:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-19T18:37:36.270 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:36.270 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:36.548 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:37 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:36.554 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados-devel 2023-12-19T18:37:36.647 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:36.700 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:36.701 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-19T18:37:36.701 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:36.757 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2023-12-19T18:37:37.120 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:32 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:37.265 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:37.266 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2023-12-19T18:37:37.266 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:37.266 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 126 k 2023-12-19T18:37:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:37.267 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:37.267 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:37.267 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2023-12-19T18:37:37.267 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:37.267 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 126 k 2023-12-19T18:37:37.267 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 448 k 2023-12-19T18:37:37.267 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:37.338 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:38 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:37.436 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:37.488 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:37.489 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-19T18:37:37.489 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:37.551 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2023-12-19T18:37:37.926 INFO:teuthology.orchestra.run.smithi089.stdout:librados-devel-17.2.7-131.ga88f3edf.el9.x86_64. 191 kB/s | 126 kB 00:00 2023-12-19T18:37:37.927 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:37.927 INFO:teuthology.orchestra.run.smithi089.stdout:Total 191 kB/s | 126 kB 00:00 2023-12-19T18:37:37.928 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:37.933 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:37.933 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:37.989 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:37.989 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:38.134 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:39 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:38.141 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:38.231 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:38.232 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:38.283 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:38.284 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-19T18:37:38.285 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:38.346 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2023-12-19T18:37:38.643 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:38.905 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:38.905 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:38.905 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:37:38.905 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:37:38.905 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:38.905 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:38.928 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:39 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:39.027 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:39.081 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:39.081 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-19T18:37:39.082 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:39.146 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2023-12-19T18:37:39.171 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs2 2023-12-19T18:37:39.729 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:40 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:39.736 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:34 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:39.825 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:39.831 INFO:teuthology.orchestra.run.smithi089.stdout:Package libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:39.879 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:39.880 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-19T18:37:39.881 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:39.883 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:39.885 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-12-19T18:37:39.885 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:39.943 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2023-12-19T18:37:39.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs-devel 2023-12-19T18:37:40.514 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:35 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:40.524 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:41 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:40.659 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 25 k 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 25 k 2023-12-19T18:37:40.660 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 110 k 2023-12-19T18:37:40.661 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:40.676 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:40.677 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:40.677 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-12-19T18:37:40.677 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:40.677 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:40.677 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 81 k 2023-12-19T18:37:40.678 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:40.678 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:40.678 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:40.678 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-19T18:37:40.678 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:40.678 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 81 k 2023-12-19T18:37:40.678 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 222 k 2023-12-19T18:37:40.678 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:41.139 INFO:teuthology.orchestra.run.smithi089.stdout:libcephfs-devel-17.2.7-131.ga88f3edf.el9.x86_64 52 kB/s | 25 kB 00:00 2023-12-19T18:37:41.140 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:41.140 INFO:teuthology.orchestra.run.smithi089.stdout:Total 52 kB/s | 25 kB 00:00 2023-12-19T18:37:41.140 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:41.143 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:41.143 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:41.145 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:41.145 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:41.233 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:41.336 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:41.419 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-17.2.7-131.ga88f3edf.el9.x86_64.rpm 110 kB/s | 81 kB 00:00 2023-12-19T18:37:41.419 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:41.420 INFO:teuthology.orchestra.run.smithi059.stdout:Total 109 kB/s | 81 kB 00:00 2023-12-19T18:37:41.420 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:41.429 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:41.430 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:41.493 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:41.493 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:41.638 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:41.722 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:41.745 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:41.988 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:37:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:37:41.988 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:41.988 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:42.125 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:42.190 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados2 2023-12-19T18:37:42.470 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:42.470 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:42.470 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:42.470 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:37:42.471 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:42.471 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:42.718 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2023-12-19T18:37:42.740 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:37 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:42.835 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:42.885 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:42.886 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-12-19T18:37:42.886 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:42.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librbd1 2023-12-19T18:37:43.309 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:44 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:43.460 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:43.461 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:43.461 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-12-19T18:37:43.461 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:43.461 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:43.461 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.9 M 2023-12-19T18:37:43.461 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:43.461 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:43.461 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:43.461 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-19T18:37:43.462 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:43.462 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.9 M 2023-12-19T18:37:43.462 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 12 M 2023-12-19T18:37:43.462 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:43.521 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:38 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:43.616 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:43.666 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:43.667 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-12-19T18:37:43.667 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:43.757 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rados 2023-12-19T18:37:44.314 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:39 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:44.407 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:44.456 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:44.457 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-12-19T18:37:44.458 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:44.545 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rgw 2023-12-19T18:37:45.098 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:40 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:45.190 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:45.240 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:45.241 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-12-19T18:37:45.241 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:45.324 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-cephfs 2023-12-19T18:37:45.876 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:40 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:45.971 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:46.023 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:46.024 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-12-19T18:37:46.024 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:46.107 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rbd 2023-12-19T18:37:46.388 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-17.2.7-131.ga88f3edf.el9.x86_64.rpm 1.0 MB/s | 2.9 MB 00:02 2023-12-19T18:37:46.389 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:46.389 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.0 MB/s | 2.9 MB 00:02 2023-12-19T18:37:46.390 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:46.401 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:46.402 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:46.481 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:46.481 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:46.663 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:41 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:46.756 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T18:37:46.805 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:46.806 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2023-12-19T18:37:46.806 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:46.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-fuse 2023-12-19T18:37:46.919 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:46.972 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:46.998 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:46.999 INFO:teuthology.orchestra.run.smithi059.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:37:46.999 INFO:teuthology.orchestra.run.smithi059.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-19T18:37:46.999 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T18:37:46.999 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T18:37:46.999 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:47.433 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:42 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 81 k 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 81 k 2023-12-19T18:37:47.579 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 222 k 2023-12-19T18:37:47.580 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:48.143 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:48.143 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:48.143 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:48.143 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:37:48.143 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:48.144 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:48.293 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-fuse-17.2.7-131.ga88f3edf.el9.x86_64.rpm 114 kB/s | 81 kB 00:00 2023-12-19T18:37:48.294 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:48.294 INFO:teuthology.orchestra.run.smithi089.stdout:Total 114 kB/s | 81 kB 00:00 2023-12-19T18:37:48.295 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:48.301 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:48.301 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:48.357 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:48.357 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:48.409 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2023-12-19T18:37:48.498 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:48.616 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:48.993 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:49 ago on Tue 19 Dec 2023 06:35:59 PM UTC. 2023-12-19T18:37:49.014 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:49.143 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-19T18:37:49.144 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:49.144 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-12-19T18:37:49.144 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:49.144 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-19T18:37:49.144 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 162 k 2023-12-19T18:37:49.144 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:49.144 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-19T18:37:49.144 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-19T18:37:49.145 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-19T18:37:49.145 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:49.145 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 162 k 2023-12-19T18:37:49.145 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 465 k 2023-12-19T18:37:49.145 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-19T18:37:49.330 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:49.330 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:49.330 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:37:49.330 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:37:49.330 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:49.331 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:37:49.594 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-mirror 2023-12-19T18:37:49.949 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-17.2.7-131.ga88f3edf.el9.x86_64.rpm 202 kB/s | 162 kB 00:00 2023-12-19T18:37:49.950 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:49.950 INFO:teuthology.orchestra.run.smithi059.stdout:Total 201 kB/s | 162 kB 00:00 2023-12-19T18:37:49.950 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-19T18:37:49.958 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-19T18:37:49.959 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-19T18:37:50.022 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-19T18:37:50.023 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-19T18:37:50.157 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:45 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:37:50.168 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-19T18:37:50.282 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:50.305 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:37:50.307 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2023-12-19T18:37:50.307 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:50.307 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:37:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.9 M 2023-12-19T18:37:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:50.308 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:37:50.308 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:37:50.308 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2023-12-19T18:37:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:37:50.308 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.9 M 2023-12-19T18:37:50.308 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 12 M 2023-12-19T18:37:50.308 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:37:50.711 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:51.068 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:51.069 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:51.069 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-19T18:37:51.069 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:37:51.069 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:37:51.070 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-19T18:37:51.290 DEBUG:teuthology.parallel:result is None 2023-12-19T18:37:59.205 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-mirror-17.2.7-131.ga88f3edf.el9.x86_64.rpm 338 kB/s | 2.9 MB 00:08 2023-12-19T18:37:59.206 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:37:59.206 INFO:teuthology.orchestra.run.smithi089.stdout:Total 338 kB/s | 2.9 MB 00:08 2023-12-19T18:37:59.206 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:37:59.216 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:37:59.217 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:37:59.291 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:37:59.292 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:37:59.707 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:37:59.755 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:59.779 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:37:59.779 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T18:37:59.779 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-19T18:37:59.779 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T18:37:59.780 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T18:37:59.780 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:38:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:38:00.827 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:38:00.828 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:38:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:38:00.828 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:38:00.828 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:38:01.110 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-nbd 2023-12-19T18:38:01.673 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:56 ago on Tue 19 Dec 2023 06:36:05 PM UTC. 2023-12-19T18:38:01.824 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2023-12-19T18:38:01.824 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:38:01.824 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2023-12-19T18:38:01.824 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:38:01.824 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2023-12-19T18:38:01.825 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 162 k 2023-12-19T18:38:01.825 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:38:01.825 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2023-12-19T18:38:01.825 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2023-12-19T18:38:01.825 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2023-12-19T18:38:01.825 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:38:01.825 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 162 k 2023-12-19T18:38:01.825 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 465 k 2023-12-19T18:38:01.825 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2023-12-19T18:38:02.826 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-nbd-17.2.7-131.ga88f3edf.el9.x86_64.rpm 162 kB/s | 162 kB 00:01 2023-12-19T18:38:02.827 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:38:02.828 INFO:teuthology.orchestra.run.smithi089.stdout:Total 162 kB/s | 162 kB 00:01 2023-12-19T18:38:02.828 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2023-12-19T18:38:02.836 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2023-12-19T18:38:02.836 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2023-12-19T18:38:02.894 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2023-12-19T18:38:02.894 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2023-12-19T18:38:03.043 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2023-12-19T18:38:03.152 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:38:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:38:03.890 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T18:38:03.890 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:38:03.890 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2023-12-19T18:38:03.890 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T18:38:03.890 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:38:03.890 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2023-12-19T18:38:04.142 DEBUG:teuthology.parallel:result is None 2023-12-19T18:38:04.142 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T18:38:04.143 INFO:teuthology.packaging:ref: None 2023-12-19T18:38:04.143 INFO:teuthology.packaging:tag: None 2023-12-19T18:38:04.143 INFO:teuthology.packaging:branch: quincy 2023-12-19T18:38:04.143 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:38:04.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-19T18:38:04.353 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-19T18:38:04.379 INFO:teuthology.orchestra.run.smithi059.stdout:17.2.7-131.ga88f3edf.el9 2023-12-19T18:38:04.380 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-131.ga88f3edf.el9 2023-12-19T18:38:04.380 INFO:teuthology.task.install:The correct ceph version 17.2.7-131.ga88f3edf is installed. 2023-12-19T18:38:04.382 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T18:38:04.382 INFO:teuthology.packaging:ref: None 2023-12-19T18:38:04.382 INFO:teuthology.packaging:tag: None 2023-12-19T18:38:04.382 INFO:teuthology.packaging:branch: quincy 2023-12-19T18:38:04.382 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:38:04.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-19T18:38:04.583 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-19T18:38:04.604 INFO:teuthology.orchestra.run.smithi089.stdout:17.2.7-131.ga88f3edf.el9 2023-12-19T18:38:04.604 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-131.ga88f3edf.el9 2023-12-19T18:38:04.604 INFO:teuthology.task.install:The correct ceph version 17.2.7-131.ga88f3edf is installed. 2023-12-19T18:38:04.605 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-19T18:38:04.606 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:38:04.606 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-19T18:38:04.642 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:38:04.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-19T18:38:04.672 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-19T18:38:04.672 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:38:04.672 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2023-12-19T18:38:04.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-19T18:38:04.789 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:38:04.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2023-12-19T18:38:04.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-19T18:38:04.881 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-19T18:38:04.882 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:38:04.882 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-19T18:38:04.915 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-19T18:38:04.991 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:38:04.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-19T18:38:05.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-19T18:38:05.084 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-19T18:38:05.085 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:38:05.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2023-12-19T18:38:05.117 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-19T18:38:05.189 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:38:05.189 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2023-12-19T18:38:05.215 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-19T18:38:05.282 INFO:teuthology.run_tasks:Running task print... 2023-12-19T18:38:05.290 INFO:teuthology.task.print:**** done install task... 2023-12-19T18:38:05.291 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-19T18:38:05.529 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-19T18:38:05.529 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2023-12-19T18:38:05.530 INFO:tasks.cephadm:Cluster fsid is bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:38:05.530 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-19T18:38:05.530 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-19T18:38:05.530 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi059': '172.21.15.59', 'mon.smithi089': '172.21.15.89'} 2023-12-19T18:38:05.530 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-19T18:38:05.530 DEBUG:teuthology.orchestra.run.smithi059:> sudo hostname $(hostname -s) 2023-12-19T18:38:05.566 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname $(hostname -s) 2023-12-19T18:38:05.594 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2023-12-19T18:38:05.594 DEBUG:teuthology.orchestra.run.smithi059:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-19T18:38:05.894 INFO:teuthology.orchestra.run.smithi059.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Dec 19 18:38 /home/ubuntu/cephtest/cephadm 2023-12-19T18:38:05.894 DEBUG:teuthology.orchestra.run.smithi089:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-19T18:38:06.064 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Dec 19 18:38 /home/ubuntu/cephtest/cephadm 2023-12-19T18:38:06.064 DEBUG:teuthology.orchestra.run.smithi059:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-19T18:38:06.084 DEBUG:teuthology.orchestra.run.smithi089:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-19T18:38:06.109 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2023-12-19T18:38:06.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2023-12-19T18:38:06.127 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2023-12-19T18:38:07.006 INFO:teuthology.orchestra.run.smithi089.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-19T18:38:07.023 INFO:teuthology.orchestra.run.smithi059.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-19T18:38:44.204 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-12-19T18:38:44.204 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)", 2023-12-19T18:38:44.204 INFO:teuthology.orchestra.run.smithi059.stdout: "image_id": "f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e", 2023-12-19T18:38:44.205 INFO:teuthology.orchestra.run.smithi059.stdout: "repo_digests": [ 2023-12-19T18:38:44.205 INFO:teuthology.orchestra.run.smithi059.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd" 2023-12-19T18:38:44.205 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2023-12-19T18:38:44.205 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-12-19T18:38:48.392 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2023-12-19T18:38:48.393 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)", 2023-12-19T18:38:48.393 INFO:teuthology.orchestra.run.smithi089.stdout: "image_id": "f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e", 2023-12-19T18:38:48.393 INFO:teuthology.orchestra.run.smithi089.stdout: "repo_digests": [ 2023-12-19T18:38:48.393 INFO:teuthology.orchestra.run.smithi089.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd" 2023-12-19T18:38:48.393 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2023-12-19T18:38:48.393 INFO:teuthology.orchestra.run.smithi089.stdout:} 2023-12-19T18:38:48.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph 2023-12-19T18:38:48.452 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph 2023-12-19T18:38:48.482 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /etc/ceph 2023-12-19T18:38:48.527 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /etc/ceph 2023-12-19T18:38:48.554 INFO:tasks.cephadm:Writing seed config... 2023-12-19T18:38:48.569 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2023-12-19T18:38:48.569 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2023-12-19T18:38:48.569 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-19T18:38:48.569 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-19T18:38:48.569 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-19T18:38:48.570 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-19T18:38:48.571 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-19T18:38:48.572 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-19T18:38:48.572 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-19T18:38:48.572 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-19T18:38:48.572 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-19T18:38:48.572 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:38:48.573 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-19T18:38:48.589 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = bf854274-9e9d-11ee-95a5-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-19T18:38:48.590 DEBUG:teuthology.orchestra.run.smithi059:mon.smithi059> sudo journalctl -f -n 0 -u ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mon.smithi059.service 2023-12-19T18:38:48.633 INFO:tasks.cephadm:Bootstrapping... 2023-12-19T18:38:48.633 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid bf854274-9e9d-11ee-95a5-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.59 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-19T18:38:48.852 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:38:48.853 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'bf854274-9e9d-11ee-95a5-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.59', '--skip-admin-label'] 2023-12-19T18:38:48.883 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 4.8.1 2023-12-19T18:38:48.884 INFO:teuthology.orchestra.run.smithi059.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-12-19T18:38:48.884 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying podman|docker is present... 2023-12-19T18:38:48.908 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 4.8.1 2023-12-19T18:38:48.908 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying lvm2 is present... 2023-12-19T18:38:48.908 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying time synchronization is in place... 2023-12-19T18:38:48.917 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-12-19T18:38:48.917 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-12-19T18:38:48.924 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-12-19T18:38:48.924 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2023-12-19T18:38:48.931 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout enabled 2023-12-19T18:38:48.938 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout active 2023-12-19T18:38:48.938 INFO:teuthology.orchestra.run.smithi059.stdout:Unit chronyd.service is enabled and running 2023-12-19T18:38:48.938 INFO:teuthology.orchestra.run.smithi059.stdout:Repeating the final host check... 2023-12-19T18:38:48.963 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout 4.8.1 2023-12-19T18:38:48.963 INFO:teuthology.orchestra.run.smithi059.stdout:podman (/bin/podman) version 4.8.1 is present 2023-12-19T18:38:48.963 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl is present 2023-12-19T18:38:48.963 INFO:teuthology.orchestra.run.smithi059.stdout:lvcreate is present 2023-12-19T18:38:48.969 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-12-19T18:38:48.969 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-12-19T18:38:48.977 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-12-19T18:38:48.977 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2023-12-19T18:38:48.984 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout enabled 2023-12-19T18:38:48.992 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout active 2023-12-19T18:38:48.992 INFO:teuthology.orchestra.run.smithi059.stdout:Unit chronyd.service is enabled and running 2023-12-19T18:38:48.992 INFO:teuthology.orchestra.run.smithi059.stdout:Host looks OK 2023-12-19T18:38:48.992 INFO:teuthology.orchestra.run.smithi059.stdout:Cluster fsid: bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:38:48.992 INFO:teuthology.orchestra.run.smithi059.stdout:Acquiring lock 140251301253184 on /run/cephadm/bf854274-9e9d-11ee-95a5-87774f69a715.lock 2023-12-19T18:38:48.992 INFO:teuthology.orchestra.run.smithi059.stdout:Lock 140251301253184 acquired on /run/cephadm/bf854274-9e9d-11ee-95a5-87774f69a715.lock 2023-12-19T18:38:48.993 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying IP 172.21.15.59 port 3300 ... 2023-12-19T18:38:48.993 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying IP 172.21.15.59 port 6789 ... 2023-12-19T18:38:48.993 INFO:teuthology.orchestra.run.smithi059.stdout:Base mon IP(s) is [172.21.15.59:3300, 172.21.15.59:6789], mon addrv is [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2023-12-19T18:38:48.996 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.59 metric 100 2023-12-19T18:38:48.997 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.59 metric 100 2023-12-19T18:38:48.999 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-12-19T18:38:49.000 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2023-12-19T18:38:49.000 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2023-12-19T18:38:49.002 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-19T18:38:49.002 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-12-19T18:38:49.003 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-12-19T18:38:49.003 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-12-19T18:38:49.003 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2023-12-19T18:38:49.003 INFO:teuthology.orchestra.run.smithi059.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-12-19T18:38:49.003 INFO:teuthology.orchestra.run.smithi059.stdout:Mon IP `172.21.15.59` is in CIDR network `172.21.0.0/20` 2023-12-19T18:38:49.003 INFO:teuthology.orchestra.run.smithi059.stdout:Mon IP `172.21.15.59` is in CIDR network `172.21.0.0/20` 2023-12-19T18:38:49.003 INFO:teuthology.orchestra.run.smithi059.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-12-19T18:38:49.004 INFO:teuthology.orchestra.run.smithi059.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-19T18:38:49.005 INFO:teuthology.orchestra.run.smithi059.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-19T18:38:49.495 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stdout f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e 2023-12-19T18:38:49.495 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-19T18:38:49.495 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Getting image source signatures 2023-12-19T18:38:49.496 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Copying blob sha256:46af8f5390d4e94fc57efb422ccb97bb53dfe5b948546bfc191b46557eb2dbd9 2023-12-19T18:38:49.496 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Copying blob sha256:7be6900e5445e78fc03bcf48e8188d376353468c6b158a72fddc62646d503826 2023-12-19T18:38:49.496 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Copying config sha256:f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e 2023-12-19T18:38:49.496 INFO:teuthology.orchestra.run.smithi059.stdout:/bin/podman: stderr Writing manifest to image destination 2023-12-19T18:38:50.459 INFO:teuthology.orchestra.run.smithi059.stdout:ceph: stdout ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable) 2023-12-19T18:38:50.459 INFO:teuthology.orchestra.run.smithi059.stdout:Ceph version: ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable) 2023-12-19T18:38:50.459 INFO:teuthology.orchestra.run.smithi059.stdout:Extracting ceph user uid/gid from container image... 2023-12-19T18:38:51.299 INFO:teuthology.orchestra.run.smithi059.stdout:stat: stdout 167 167 2023-12-19T18:38:51.299 INFO:teuthology.orchestra.run.smithi059.stdout:Creating initial keys... 2023-12-19T18:38:52.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph-authtool: stdout AQA744FlbZC3IRAAq8DOcUJdbRiyrSc/PHdhZg== 2023-12-19T18:38:53.060 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph-authtool: stdout AQA844FleWYgIxAAqQ2SJeyGwO93INTL9lIw5A== 2023-12-19T18:38:53.924 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph-authtool: stdout AQA944FlkbZwFRAAVzI4jnUEevFqQW+Kx2kj4Q== 2023-12-19T18:38:53.925 INFO:teuthology.orchestra.run.smithi059.stdout:Creating initial monmap... 2023-12-19T18:38:54.956 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-19T18:38:54.956 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2023-12-19T18:38:54.956 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:38:54.956 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-19T18:38:54.957 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool for smithi059 [v2:172.21.15.59:3300,v1:172.21.15.59:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-19T18:38:54.957 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = octopus 2023-12-19T18:38:54.957 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: set fsid to bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:38:54.957 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-19T18:38:54.957 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:38:54.957 INFO:teuthology.orchestra.run.smithi059.stdout:Creating mon... 2023-12-19T18:38:55.810 INFO:teuthology.orchestra.run.smithi059.stdout:create mon.smithi059 on 2023-12-19T18:38:56.010 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2023-12-19T18:38:56.161 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-19T18:38:56.381 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-bf854274-9e9d-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-bf854274-9e9d-11ee-95a5-87774f69a715.target. 2023-12-19T18:38:56.381 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-bf854274-9e9d-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-bf854274-9e9d-11ee-95a5-87774f69a715.target. 2023-12-19T18:38:56.552 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mon.smithi059 2023-12-19T18:38:56.552 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mon.smithi059.service: Unit ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mon.smithi059.service not loaded. 2023-12-19T18:38:56.712 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bf854274-9e9d-11ee-95a5-87774f69a715.target.wants/ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mon.smithi059.service → /etc/systemd/system/ceph-bf854274-9e9d-11ee-95a5-87774f69a715@.service. 2023-12-19T18:38:57.168 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:57 smithi059 podman[80273]: 2023-12-19 18:38:57.107090167 +0000 UTC m=+0.280119055 container start a78e5f51fa33356294b0edb590c8cca263ad42ddd16cb194596aba53ca3a3ea9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-a88f3ed) 2023-12-19T18:38:57.168 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:57 smithi059 ceph-mon[80292]: mkfs bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:38:57.224 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-19T18:38:57.224 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2023-12-19T18:38:57.232 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-19T18:38:57.232 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2023-12-19T18:38:57.232 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2023-12-19T18:38:57.232 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to enable service . firewalld.service is not available 2023-12-19T18:38:57.233 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mon to start... 2023-12-19T18:38:57.233 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mon... 2023-12-19T18:38:57.421 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:57 smithi059 ceph-mon[80292]: mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2023-12-19T18:38:57.421 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:57 smithi059 bash[80273]: a78e5f51fa33356294b0edb590c8cca263ad42ddd16cb194596aba53ca3a3ea9 2023-12-19T18:38:57.421 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:57 smithi059 systemd[1]: Started Ceph mon.smithi059 for bf854274-9e9d-11ee-95a5-87774f69a715. 2023-12-19T18:38:58.275 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:58 smithi059 ceph-mon[80292]: mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2023-12-19T18:38:58.276 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:58 smithi059 ceph-mon[80292]: monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-19T18:38:58.276 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:58 smithi059 ceph-mon[80292]: fsmap 2023-12-19T18:38:58.276 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:58 smithi059 ceph-mon[80292]: osdmap e1: 0 total, 0 up, 0 in 2023-12-19T18:38:58.276 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:58 smithi059 ceph-mon[80292]: mgrmap e1: no daemons active 2023-12-19T18:38:58.276 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:58 smithi059 ceph-mon[80292]: from='client.? 172.21.15.59:0/1616072692' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-19T18:38:58.285 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout cluster: 2023-12-19T18:38:58.286 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout id: bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:38:58.286 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-12-19T18:38:58.286 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:38:58.286 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout services: 2023-12-19T18:38:58.286 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi059 (age 0.538037s) 2023-12-19T18:38:58.286 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-12-19T18:38:58.286 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-12-19T18:38:58.286 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:38:58.287 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout data: 2023-12-19T18:38:58.287 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-12-19T18:38:58.287 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-12-19T18:38:58.287 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-12-19T18:38:58.287 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout pgs: 2023-12-19T18:38:58.287 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:38:58.287 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.622+0000 7f0e26847700 1 Processor -- start 2023-12-19T18:38:58.287 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.622+0000 7f0e26847700 1 -- start start 2023-12-19T18:38:58.287 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.622+0000 7f0e26847700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e201015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:38:58.288 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.622+0000 7f0e26847700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e20101b20 con 0x7f0e201011d0 2023-12-19T18:38:58.288 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.622+0000 7f0e1ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e201015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:38:58.288 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.622+0000 7f0e1ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e201015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49238/0 (socket says 172.21.15.59:49238) 2023-12-19T18:38:58.288 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.622+0000 7f0e1ffff700 1 -- 172.21.15.59:0/2793556287 learned_addr learned my addr 172.21.15.59:0/2793556287 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:38:58.288 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.623+0000 7f0e1ffff700 1 -- 172.21.15.59:0/2793556287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e20101c60 con 0x7f0e201011d0 2023-12-19T18:38:58.288 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.623+0000 7f0e1ffff700 1 --2- 172.21.15.59:0/2793556287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e201015e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000bb80 tx=0x7f0e1000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb1a4edf0c2dc54 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:38:58.288 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.623+0000 7f0e1effd700 1 -- 172.21.15.59:0/2793556287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e1000dd00 con 0x7f0e201011d0 2023-12-19T18:38:58.288 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.623+0000 7f0e1effd700 1 -- 172.21.15.59:0/2793556287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1000de60 con 0x7f0e201011d0 2023-12-19T18:38:58.289 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.623+0000 7f0e1effd700 1 -- 172.21.15.59:0/2793556287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e100105f0 con 0x7f0e201011d0 2023-12-19T18:38:58.289 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.623+0000 7f0e26847700 1 -- 172.21.15.59:0/2793556287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 msgr2=0x7f0e201015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:38:58.289 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.623+0000 7f0e26847700 1 --2- 172.21.15.59:0/2793556287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e201015e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000bb80 tx=0x7f0e1000b4f0 comp rx=0 tx=0).stop 2023-12-19T18:38:58.289 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.624+0000 7f0e26847700 1 -- 172.21.15.59:0/2793556287 shutdown_connections 2023-12-19T18:38:58.289 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.624+0000 7f0e26847700 1 --2- 172.21.15.59:0/2793556287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e201015e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:38:58.289 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.624+0000 7f0e26847700 1 -- 172.21.15.59:0/2793556287 >> 172.21.15.59:0/2793556287 conn(0x7f0e200fc950 msgr2=0x7f0e200fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:38:58.289 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.624+0000 7f0e26847700 1 -- 172.21.15.59:0/2793556287 shutdown_connections 2023-12-19T18:38:58.289 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.624+0000 7f0e26847700 1 -- 172.21.15.59:0/2793556287 wait complete. 2023-12-19T18:38:58.289 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.624+0000 7f0e26847700 1 Processor -- start 2023-12-19T18:38:58.290 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.624+0000 7f0e26847700 1 -- start start 2023-12-19T18:38:58.290 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.624+0000 7f0e26847700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e2010e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:38:58.290 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.624+0000 7f0e26847700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2010ecd0 con 0x7f0e201011d0 2023-12-19T18:38:58.290 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e1ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e2010e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:38:58.290 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e1ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e2010e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49254/0 (socket says 172.21.15.59:49254) 2023-12-19T18:38:58.290 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e1ffff700 1 -- 172.21.15.59:0/1616072692 learned_addr learned my addr 172.21.15.59:0/1616072692 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:38:58.290 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e1ffff700 1 -- 172.21.15.59:0/1616072692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1000b790 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e1ffff700 1 --2- 172.21.15.59:0/1616072692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e2010e790 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000b530 tx=0x7f0e1001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e1cff9700 1 -- 172.21.15.59:0/1616072692 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e100189b0 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e1cff9700 1 -- 172.21.15.59:0/1616072692 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1001d070 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e26847700 1 -- 172.21.15.59:0/1616072692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2010eed0 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e1cff9700 1 -- 172.21.15.59:0/1616072692 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e10018cb0 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.625+0000 7f0e26847700 1 -- 172.21.15.59:0/1616072692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2010f2e0 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.626+0000 7f0e1cff9700 1 -- 172.21.15.59:0/1616072692 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0e1002e460 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.626+0000 7f0e1cff9700 1 -- 172.21.15.59:0/1616072692 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0e10014070 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.626+0000 7f0e26847700 1 -- 172.21.15.59:0/1616072692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2004bb80 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.629+0000 7f0e1cff9700 1 -- 172.21.15.59:0/1616072692 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0e1001f050 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.703+0000 7f0e26847700 1 -- 172.21.15.59:0/1616072692 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f0e20107a60 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.703+0000 7f0e1cff9700 1 -- 172.21.15.59:0/1616072692 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f0e10018e60 con 0x7f0e201011d0 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.704+0000 7f0e26847700 1 -- 172.21.15.59:0/1616072692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 msgr2=0x7f0e2010e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:38:58.291 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.704+0000 7f0e26847700 1 --2- 172.21.15.59:0/1616072692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e2010e790 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0e1000b530 tx=0x7f0e1001a000 comp rx=0 tx=0).stop 2023-12-19T18:38:58.292 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.704+0000 7f0e26847700 1 -- 172.21.15.59:0/1616072692 shutdown_connections 2023-12-19T18:38:58.292 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.704+0000 7f0e26847700 1 --2- 172.21.15.59:0/1616072692 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e201011d0 0x7f0e2010e790 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:38:58.292 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.704+0000 7f0e26847700 1 -- 172.21.15.59:0/1616072692 >> 172.21.15.59:0/1616072692 conn(0x7f0e200fc950 msgr2=0x7f0e200fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:38:58.292 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.704+0000 7f0e26847700 1 -- 172.21.15.59:0/1616072692 shutdown_connections 2023-12-19T18:38:58.292 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:57.704+0000 7f0e26847700 1 -- 172.21.15.59:0/1616072692 wait complete. 2023-12-19T18:38:58.292 INFO:teuthology.orchestra.run.smithi059.stdout:mon is available 2023-12-19T18:38:58.292 INFO:teuthology.orchestra.run.smithi059.stdout:Assimilating anything we can from ceph.conf... 2023-12-19T18:38:59.266 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:38:59.266 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [global] 2023-12-19T18:38:59.266 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout fsid = bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:38:59.266 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2023-12-19T18:38:59.266 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-12-19T18:38:59.266 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-12-19T18:38:59.266 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-12-19T18:38:59.266 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-12-19T18:38:59.266 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [mgr] 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [osd] 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.697+0000 7f0ba8f1f700 1 Processor -- start 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.697+0000 7f0ba8f1f700 1 -- start start 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.697+0000 7f0ba8f1f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.697+0000 7f0ba8f1f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba4104310 con 0x7f0ba41039c0 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.697+0000 7f0ba259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:38:59.267 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.697+0000 7f0ba259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49264/0 (socket says 172.21.15.59:49264) 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.697+0000 7f0ba259c700 1 -- 172.21.15.59:0/3995631859 learned_addr learned my addr 172.21.15.59:0/3995631859 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.698+0000 7f0ba259c700 1 -- 172.21.15.59:0/3995631859 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba4101fb0 con 0x7f0ba41039c0 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.698+0000 7f0ba259c700 1 --2- 172.21.15.59:0/3995631859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4103dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b940098e0 tx=0x7f0b94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72c4f1a5f4cda5d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.698+0000 7f0ba159a700 1 -- 172.21.15.59:0/3995631859 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9400b860 con 0x7f0ba41039c0 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.698+0000 7f0ba159a700 1 -- 172.21.15.59:0/3995631859 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9400b9c0 con 0x7f0ba41039c0 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.698+0000 7f0ba159a700 1 -- 172.21.15.59:0/3995631859 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9400bc90 con 0x7f0ba41039c0 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.698+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3995631859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 msgr2=0x7f0ba4103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.698+0000 7f0ba8f1f700 1 --2- 172.21.15.59:0/3995631859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4103dd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0b940098e0 tx=0x7f0b94009250 comp rx=0 tx=0).stop 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3995631859 shutdown_connections 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba8f1f700 1 --2- 172.21.15.59:0/3995631859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4103dd0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3995631859 >> 172.21.15.59:0/3995631859 conn(0x7f0ba40fda40 msgr2=0x7f0ba40ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3995631859 shutdown_connections 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3995631859 wait complete. 2023-12-19T18:38:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba8f1f700 1 Processor -- start 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba8f1f700 1 -- start start 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba8f1f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4110990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba8f1f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba4110ed0 con 0x7f0ba41039c0 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4110990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.699+0000 7f0ba259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4110990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49272/0 (socket says 172.21.15.59:49272) 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0ba259c700 1 -- 172.21.15.59:0/3562251548 learned_addr learned my addr 172.21.15.59:0/3562251548 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0ba259c700 1 -- 172.21.15.59:0/3562251548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b940094f0 con 0x7f0ba41039c0 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0ba259c700 1 --2- 172.21.15.59:0/3562251548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4110990 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0b94003080 tx=0x7f0b94016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0b92ffd700 1 -- 172.21.15.59:0/3562251548 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9401c5c0 con 0x7f0ba41039c0 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0b92ffd700 1 -- 172.21.15.59:0/3562251548 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9401c720 con 0x7f0ba41039c0 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3562251548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba41110d0 con 0x7f0ba41039c0 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0b92ffd700 1 -- 172.21.15.59:0/3562251548 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9401ca30 con 0x7f0ba41039c0 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3562251548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba41114e0 con 0x7f0ba41039c0 2023-12-19T18:38:59.269 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0b92ffd700 1 -- 172.21.15.59:0/3562251548 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0b9401a450 con 0x7f0ba41039c0 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.700+0000 7f0b92ffd700 1 -- 172.21.15.59:0/3562251548 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0b9402e790 con 0x7f0ba41039c0 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.701+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3562251548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba41022c0 con 0x7f0ba41039c0 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.703+0000 7f0b92ffd700 1 -- 172.21.15.59:0/3562251548 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0b9401a890 con 0x7f0ba41039c0 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.774+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3562251548 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0ba404bb80 con 0x7f0ba41039c0 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.777+0000 7f0b92ffd700 1 -- 172.21.15.59:0/3562251548 <== mon.0 v2:172.21.15.59:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0b9402ee60 con 0x7f0ba41039c0 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.777+0000 7f0b92ffd700 1 -- 172.21.15.59:0/3562251548 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f0b9401b730 con 0x7f0ba41039c0 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.778+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3562251548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 msgr2=0x7f0ba4110990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.778+0000 7f0ba8f1f700 1 --2- 172.21.15.59:0/3562251548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4110990 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0b94003080 tx=0x7f0b94016000 comp rx=0 tx=0).stop 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.778+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3562251548 shutdown_connections 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.778+0000 7f0ba8f1f700 1 --2- 172.21.15.59:0/3562251548 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0ba41039c0 0x7f0ba4110990 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.778+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3562251548 >> 172.21.15.59:0/3562251548 conn(0x7f0ba40fda40 msgr2=0x7f0ba4105040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.778+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3562251548 shutdown_connections 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:58.778+0000 7f0ba8f1f700 1 -- 172.21.15.59:0/3562251548 wait complete. 2023-12-19T18:38:59.270 INFO:teuthology.orchestra.run.smithi059.stdout:Generating new minimal ceph.conf... 2023-12-19T18:38:59.366 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:59 smithi059 ceph-mon[80292]: from='client.? 172.21.15.59:0/3562251548' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-19T18:38:59.366 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:38:59 smithi059 ceph-mon[80292]: from='client.? 172.21.15.59:0/3562251548' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-19T18:39:00.271 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:00 smithi059 ceph-mon[80292]: from='client.? 172.21.15.59:0/2404941751' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:39:00.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.666+0000 7f8081157700 1 Processor -- start 2023-12-19T18:39:00.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.666+0000 7f8081157700 1 -- start start 2023-12-19T18:39:00.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.666+0000 7f8081157700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:00.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.666+0000 7f8081157700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c104310 con 0x7f807c1039c0 2023-12-19T18:39:00.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.667+0000 7f807ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:00.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.667+0000 7f807ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49282/0 (socket says 172.21.15.59:49282) 2023-12-19T18:39:00.278 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.667+0000 7f807ad9d700 1 -- 172.21.15.59:0/4035377700 learned_addr learned my addr 172.21.15.59:0/4035377700 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.667+0000 7f807ad9d700 1 -- 172.21.15.59:0/4035377700 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c101fb0 con 0x7f807c1039c0 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.667+0000 7f807ad9d700 1 --2- 172.21.15.59:0/4035377700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f80700098e0 tx=0x7f8070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4876d137c25a5b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.667+0000 7f8079d9b700 1 -- 172.21.15.59:0/4035377700 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f807000b860 con 0x7f807c1039c0 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.667+0000 7f8079d9b700 1 -- 172.21.15.59:0/4035377700 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f807000be30 con 0x7f807c1039c0 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.667+0000 7f8079d9b700 1 -- 172.21.15.59:0/4035377700 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8070010380 con 0x7f807c1039c0 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.668+0000 7f8081157700 1 -- 172.21.15.59:0/4035377700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 msgr2=0x7f807c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.668+0000 7f8081157700 1 --2- 172.21.15.59:0/4035377700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f80700098e0 tx=0x7f8070009250 comp rx=0 tx=0).stop 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.668+0000 7f8081157700 1 -- 172.21.15.59:0/4035377700 shutdown_connections 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.668+0000 7f8081157700 1 --2- 172.21.15.59:0/4035377700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c103dd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.668+0000 7f8081157700 1 -- 172.21.15.59:0/4035377700 >> 172.21.15.59:0/4035377700 conn(0x7f807c0fda40 msgr2=0x7f807c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.668+0000 7f8081157700 1 -- 172.21.15.59:0/4035377700 shutdown_connections 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.668+0000 7f8081157700 1 -- 172.21.15.59:0/4035377700 wait complete. 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f8081157700 1 Processor -- start 2023-12-19T18:39:00.279 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f8081157700 1 -- start start 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f8081157700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c1110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f8081157700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c111630 con 0x7f807c1039c0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f807ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c1110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f807ad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c1110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49290/0 (socket says 172.21.15.59:49290) 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f807ad9d700 1 -- 172.21.15.59:0/2404941751 learned_addr learned my addr 172.21.15.59:0/2404941751 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f807ad9d700 1 -- 172.21.15.59:0/2404941751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80700094f0 con 0x7f807c1039c0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f807ad9d700 1 --2- 172.21.15.59:0/2404941751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c1110f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8070006b00 tx=0x7f8070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.669+0000 7f80637fe700 1 -- 172.21.15.59:0/2404941751 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80700109b0 con 0x7f807c1039c0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.670+0000 7f80637fe700 1 -- 172.21.15.59:0/2404941751 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f807001a430 con 0x7f807c1039c0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.670+0000 7f80637fe700 1 -- 172.21.15.59:0/2404941751 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80700245b0 con 0x7f807c1039c0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.670+0000 7f8081157700 1 -- 172.21.15.59:0/2404941751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c111830 con 0x7f807c1039c0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.670+0000 7f8081157700 1 -- 172.21.15.59:0/2404941751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c111c40 con 0x7f807c1039c0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.670+0000 7f80637fe700 1 -- 172.21.15.59:0/2404941751 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8070024ab0 con 0x7f807c1039c0 2023-12-19T18:39:00.280 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.670+0000 7f80637fe700 1 -- 172.21.15.59:0/2404941751 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f807002eb90 con 0x7f807c1039c0 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.671+0000 7f8081157700 1 -- 172.21.15.59:0/2404941751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c102d60 con 0x7f807c1039c0 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.673+0000 7f80637fe700 1 -- 172.21.15.59:0/2404941751 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f807002eda0 con 0x7f807c1039c0 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.744+0000 7f8081157700 1 -- 172.21.15.59:0/2404941751 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f807c10ad00 con 0x7f807c1039c0 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.744+0000 7f80637fe700 1 -- 172.21.15.59:0/2404941751 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f807001a9c0 con 0x7f807c1039c0 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.745+0000 7f8081157700 1 -- 172.21.15.59:0/2404941751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 msgr2=0x7f807c1110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.745+0000 7f8081157700 1 --2- 172.21.15.59:0/2404941751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c1110f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8070006b00 tx=0x7f8070009250 comp rx=0 tx=0).stop 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.745+0000 7f8081157700 1 -- 172.21.15.59:0/2404941751 shutdown_connections 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.745+0000 7f8081157700 1 --2- 172.21.15.59:0/2404941751 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f807c1039c0 0x7f807c1110f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.745+0000 7f8081157700 1 -- 172.21.15.59:0/2404941751 >> 172.21.15.59:0/2404941751 conn(0x7f807c0fda40 msgr2=0x7f807c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.745+0000 7f8081157700 1 -- 172.21.15.59:0/2404941751 shutdown_connections 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:38:59.745+0000 7f8081157700 1 -- 172.21.15.59:0/2404941751 wait complete. 2023-12-19T18:39:00.281 INFO:teuthology.orchestra.run.smithi059.stdout:Restarting the monitor... 2023-12-19T18:39:00.525 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:00 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for bf854274-9e9d-11ee-95a5-87774f69a715... 2023-12-19T18:39:00.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80288]: 2023-12-19T18:39:00.525+0000 7f73ac57e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T18:39:00.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80288]: 2023-12-19T18:39:00.525+0000 7f73ac57e700 -1 mon.smithi059@0(leader) e1 *** Got Signal Terminated *** 2023-12-19T18:39:00.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:00 smithi059 podman[80535]: 2023-12-19 18:39:00.751583191 +0000 UTC m=+0.321917427 container died a78e5f51fa33356294b0edb590c8cca263ad42ddd16cb194596aba53ca3a3ea9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, GIT_CLEAN=True, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, ceph=True) 2023-12-19T18:39:01.233 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 podman[80535]: 2023-12-19 18:39:01.025124431 +0000 UTC m=+0.595458648 container cleanup a78e5f51fa33356294b0edb590c8cca263ad42ddd16cb194596aba53ca3a3ea9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=) 2023-12-19T18:39:01.233 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 bash[80535]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059 2023-12-19T18:39:01.568 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 podman[80547]: 2023-12-19 18:39:01.234298375 +0000 UTC m=+0.482963057 container remove a78e5f51fa33356294b0edb590c8cca263ad42ddd16cb194596aba53ca3a3ea9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2023-12-19T18:39:01.568 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 systemd[1]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mon.smithi059.service: Deactivated successfully. 2023-12-19T18:39:01.568 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 systemd[1]: Stopped Ceph mon.smithi059 for bf854274-9e9d-11ee-95a5-87774f69a715. 2023-12-19T18:39:01.568 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 systemd[1]: Starting Ceph mon.smithi059 for bf854274-9e9d-11ee-95a5-87774f69a715... 2023-12-19T18:39:01.819 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 podman[80642]: 2023-12-19 18:39:01.471616524 +0000 UTC m=+0.018584375 image pull f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e quay.ceph.io/ceph-ci/ceph:quincy 2023-12-19T18:39:01.820 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 podman[80642]: 2023-12-19 18:39:01.601880042 +0000 UTC m=+0.148847887 container create 297c7051711f656f92e79be4daf1e761c5efbcb116487e3e7a0307c567c60f4d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, RELEASE=quincy-a88f3ed) 2023-12-19T18:39:01.820 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 podman[80642]: 2023-12-19 18:39:01.820771705 +0000 UTC m=+0.367739549 container init 297c7051711f656f92e79be4daf1e761c5efbcb116487e3e7a0307c567c60f4d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, RELEASE=quincy-a88f3ed, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20231212) 2023-12-19T18:39:01.843 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 podman[80642]: 2023-12-19 18:39:01.823899653 +0000 UTC m=+0.370867503 container start 297c7051711f656f92e79be4daf1e761c5efbcb116487e3e7a0307c567c60f4d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, maintainer=Guillaume Abrioux , GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-a88f3ed, io.buildah.version=1.31.3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2023-12-19T18:39:01.843 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: set uid:gid to 167:167 (ceph:ceph) 2023-12-19T18:39:01.844 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable), process ceph-mon, pid 2 2023-12-19T18:39:01.845 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: pidfile_write: ignore empty --pid-file 2023-12-19T18:39:01.940 INFO:teuthology.orchestra.run.smithi059.stdout:Setting mon public_network to 172.21.0.0/20 2023-12-19T18:39:02.123 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: load: jerasure load: lrc 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: RocksDB version: 6.15.5 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Compile date Dec 19 2023 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: DB SUMMARY 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: DB Session ID: C6325JV40X4QO7BLL8IO 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: CURRENT file: CURRENT 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: IDENTITY file: IDENTITY 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi059/store.db dir, Total Num: 1, files: 000008.sst 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi059/store.db: 000010.log size: 96764 ; 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.error_if_exists: 0 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.create_if_missing: 0 2023-12-19T18:39:02.124 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.paranoid_checks: 1 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.env: 0x556469f61a80 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.fs: Posix File System 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.info_log: 0x55646bded000 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_file_opening_threads: 16 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.statistics: (nil) 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.use_fsync: 0 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_log_file_size: 0 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.log_file_time_to_roll: 0 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.keep_log_file_num: 1000 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.recycle_log_file_num: 0 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.allow_fallocate: 1 2023-12-19T18:39:02.125 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.allow_mmap_reads: 0 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.allow_mmap_writes: 0 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.use_direct_reads: 0 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.create_missing_column_families: 0 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.db_log_dir: 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi059/store.db 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.table_cache_numshardbits: 6 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.WAL_ttl_seconds: 0 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.WAL_size_limit_MB: 0 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-19T18:39:02.126 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.is_fd_close_on_exec: 1 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.advise_random_on_open: 1 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.db_write_buffer_size: 0 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.write_buffer_manager: 0x55646cb629f0 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.use_adaptive_mutex: 0 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.rate_limiter: (nil) 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.wal_recovery_mode: 2 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.enable_thread_tracking: 0 2023-12-19T18:39:02.128 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.enable_pipelined_write: 0 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.unordered_write: 0 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.row_cache: None 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.wal_filter: None 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.allow_ingest_behind: 0 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.preserve_deletes: 0 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.two_write_queues: 0 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.manual_wal_flush: 0 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.atomic_flush: 0 2023-12-19T18:39:02.129 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.persist_stats_to_disk: 0 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.write_dbid_to_manifest: 0 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.log_readahead_size: 0 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.best_efforts_recovery: 0 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.allow_data_in_errors: 0 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.db_host_id: __hostname__ 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_background_jobs: 2 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_background_compactions: -1 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_subcompactions: 1 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-19T18:39:02.130 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.delayed_write_rate : 16777216 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_total_wal_size: 0 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.stats_dump_period_sec: 600 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.stats_persist_period_sec: 600 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_open_files: -1 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bytes_per_sync: 0 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.wal_bytes_per_sync: 0 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.strict_bytes_per_sync: 0 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_readahead_size: 0 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_background_flushes: -1 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Compression algorithms supported: 2023-12-19T18:39:02.131 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: kZSTD supported: 0 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: kXpressCompression supported: 0 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: kLZ4HCCompression supported: 1 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: kZlibCompression supported: 1 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: kSnappyCompression supported: 1 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: kLZ4Compression supported: 1 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: kBZip2Compression supported: 0 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Fast CRC32 supported: Supported on x86 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000009 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.merge_operator: 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_filter: None 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_filter_factory: None 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.sst_partitioner_factory: None 2023-12-19T18:39:02.132 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.memtable_factory: SkipListFactory 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.table_factory: BlockBasedTable 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55646be42f90) 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: cache_index_and_filter_blocks: 1 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: pin_top_level_index_and_filter: 1 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: index_type: 0 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: data_block_index_type: 0 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: index_shortening: 1 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: data_block_hash_table_util_ratio: 0.750000 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: hash_index_allow_collision: 1 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: checksum: 1 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: no_block_cache: 0 2023-12-19T18:39:02.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache: 0x55646be75090 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache_name: BinnedLRUCache 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache_options: 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: capacity : 536870912 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: num_shard_bits : 4 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: strict_capacity_limit : 0 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: high_pri_pool_ratio: 0.000 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_cache_compressed: (nil) 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: persistent_cache: (nil) 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_size: 4096 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_size_deviation: 10 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_restart_interval: 16 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: index_block_restart_interval: 1 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: metadata_block_size: 4096 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: partition_filters: 0 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: use_delta_encoding: 1 2023-12-19T18:39:02.134 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: whole_key_filtering: 1 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: verify_compression: 0 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: read_amp_bytes_per_bit: 0 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: format_version: 4 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: enable_index_compression: 1 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: block_align: 0 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.write_buffer_size: 33554432 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_write_buffer_number: 2 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compression: NoCompression 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bottommost_compression: Disabled 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.prefix_extractor: nullptr 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.num_levels: 7 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-19T18:39:02.135 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compression_opts.window_bits: -14 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compression_opts.level: 32767 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compression_opts.strategy: 0 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compression_opts.enabled: false 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-19T18:39:02.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.target_file_size_base: 67108864 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.target_file_size_multiplier: 1 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-19T18:39:02.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.arena_block_size: 4194304 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.disable_auto_compactions: 0 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-19T18:39:02.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.table_properties_collectors: 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.inplace_update_support: 0 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.inplace_update_num_locks: 10000 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.memtable_huge_page_size: 0 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.bloom_locality: 0 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.max_successive_merges: 0 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.optimize_filters_for_hits: 0 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.paranoid_file_checks: 0 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.force_consistency_checks: 1 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.report_bg_io_stats: 0 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.ttl: 2592000 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.periodic_compaction_seconds: 0 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.enable_blob_files: false 2023-12-19T18:39:02.139 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.min_blob_size: 0 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.blob_file_size: 268435456 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.blob_compression_type: NoCompression 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.enable_blob_garbage_collection: false 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: EVENT_LOG_v1 {"time_micros": 1703011141850642, "job": 1, "event": "recovery_started", "wal_files": [10]} 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: EVENT_LOG_v1 {"time_micros": 1703011141851490, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92133, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90228, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 82967, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1703011141, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "dd1baf98-d204-47f0-b7d6-ed757b78f848", "db_session_id": "C6325JV40X4QO7BLL8IO"}} 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: EVENT_LOG_v1 {"time_micros": 1703011141851862, "job": 1, "event": "recovery_finished"} 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi059/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-19T18:39:02.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55646be09180 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: DB pointer 0x55646bede000 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: rocksdb: [db/db_impl/db_impl.cc:903] 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** DB Stats ** 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-19T18:39:02.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: L0 2/0 91.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.001 0 0 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Sum 2/0 91.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.001 0 0 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.001 0 0 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.9 0.00 0.00 1 0.001 0 0 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-19T18:39:02.142 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative compaction: 0.00 GB write, 25.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval compaction: 0.00 GB write, 25.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: L0 2/0 91.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.001 0 0 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Sum 2/0 91.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.9 0.00 0.00 1 0.001 0 0 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** Compaction Stats [default] ** 2023-12-19T18:39:02.143 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.9 0.00 0.00 1 0.001 0 0 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Cumulative compaction: 0.00 GB write, 25.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: starting mon.smithi059 rank 0 at public addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] at bind addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi059 fsid bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:39:02.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mon.smithi059@-1(???) e1 preinit fsid bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mon.smithi059@-1(???).mds e1 new map 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mon.smithi059@-1(???).mds e1 print_map 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: e1 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: legacy client fscid: -1 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout: No filesystems configured 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mon.smithi059@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mon.smithi059@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-19T18:39:02.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2023-12-19T18:39:02.146 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-19T18:39:02.146 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: fsmap 2023-12-19T18:39:02.146 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: osdmap e1: 0 total, 0 up, 0 in 2023-12-19T18:39:02.146 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 ceph-mon[80661]: mgrmap e1: no daemons active 2023-12-19T18:39:02.146 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 bash[80642]: 297c7051711f656f92e79be4daf1e761c5efbcb116487e3e7a0307c567c60f4d 2023-12-19T18:39:02.146 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:01 smithi059 systemd[1]: Started Ceph mon.smithi059 for bf854274-9e9d-11ee-95a5-87774f69a715. 2023-12-19T18:39:02.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.350+0000 7f8f4e10d700 1 Processor -- start 2023-12-19T18:39:02.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.350+0000 7f8f4e10d700 1 -- start start 2023-12-19T18:39:02.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.350+0000 7f8f4e10d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:02.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.350+0000 7f8f4e10d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48103230 con 0x7f8f481028e0 2023-12-19T18:39:02.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.350+0000 7f8f477fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:02.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.350+0000 7f8f477fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49524/0 (socket says 172.21.15.59:49524) 2023-12-19T18:39:02.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.351+0000 7f8f477fe700 1 -- 172.21.15.59:0/3139611114 learned_addr learned my addr 172.21.15.59:0/3139611114 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.351+0000 7f8f477fe700 1 -- 172.21.15.59:0/3139611114 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f48103370 con 0x7f8f481028e0 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.351+0000 7f8f477fe700 1 --2- 172.21.15.59:0/3139611114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48102cf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8f380098e0 tx=0x7f8f38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0abaf3d6c7868 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.351+0000 7f8f467fc700 1 -- 172.21.15.59:0/3139611114 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f3800b8f0 con 0x7f8f481028e0 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.351+0000 7f8f467fc700 1 -- 172.21.15.59:0/3139611114 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f8f38011420 con 0x7f8f481028e0 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.351+0000 7f8f467fc700 1 -- 172.21.15.59:0/3139611114 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f38010540 con 0x7f8f481028e0 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.352+0000 7f8f4e10d700 1 -- 172.21.15.59:0/3139611114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 msgr2=0x7f8f48102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.352+0000 7f8f4e10d700 1 --2- 172.21.15.59:0/3139611114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48102cf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8f380098e0 tx=0x7f8f38009250 comp rx=0 tx=0).stop 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.352+0000 7f8f4e10d700 1 -- 172.21.15.59:0/3139611114 shutdown_connections 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.352+0000 7f8f4e10d700 1 --2- 172.21.15.59:0/3139611114 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48102cf0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.352+0000 7f8f4e10d700 1 -- 172.21.15.59:0/3139611114 >> 172.21.15.59:0/3139611114 conn(0x7f8f480fc0b0 msgr2=0x7f8f480fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:02.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.352+0000 7f8f4e10d700 1 -- 172.21.15.59:0/3139611114 shutdown_connections 2023-12-19T18:39:02.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.352+0000 7f8f4e10d700 1 -- 172.21.15.59:0/3139611114 wait complete. 2023-12-19T18:39:02.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.352+0000 7f8f4e10d700 1 Processor -- start 2023-12-19T18:39:02.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.353+0000 7f8f4e10d700 1 -- start start 2023-12-19T18:39:02.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.353+0000 7f8f4e10d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48113400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:02.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.353+0000 7f8f4e10d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48113940 con 0x7f8f481028e0 2023-12-19T18:39:02.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.353+0000 7f8f477fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48113400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:02.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.353+0000 7f8f477fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48113400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49540/0 (socket says 172.21.15.59:49540) 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.353+0000 7f8f477fe700 1 -- 172.21.15.59:0/2201068580 learned_addr learned my addr 172.21.15.59:0/2201068580 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.353+0000 7f8f477fe700 1 -- 172.21.15.59:0/2201068580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f380094f0 con 0x7f8f481028e0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.353+0000 7f8f477fe700 1 --2- 172.21.15.59:0/2201068580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48113400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8f38011d20 tx=0x7f8f38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.354+0000 7f8f2ffff700 1 -- 172.21.15.59:0/2201068580 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f3800be80 con 0x7f8f481028e0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.354+0000 7f8f2ffff700 1 -- 172.21.15.59:0/2201068580 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f8f38010e80 con 0x7f8f481028e0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.354+0000 7f8f4e10d700 1 -- 172.21.15.59:0/2201068580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f48113b40 con 0x7f8f481028e0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.354+0000 7f8f2ffff700 1 -- 172.21.15.59:0/2201068580 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f3801ae80 con 0x7f8f481028e0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.354+0000 7f8f4e10d700 1 -- 172.21.15.59:0/2201068580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f48113f50 con 0x7f8f481028e0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.354+0000 7f8f2ffff700 1 -- 172.21.15.59:0/2201068580 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8f38024740 con 0x7f8f481028e0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.354+0000 7f8f2ffff700 1 -- 172.21.15.59:0/2201068580 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8f3802ea40 con 0x7f8f481028e0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.355+0000 7f8f4e10d700 1 -- 172.21.15.59:0/2201068580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4804bb80 con 0x7f8f481028e0 2023-12-19T18:39:02.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.357+0000 7f8f2ffff700 1 -- 172.21.15.59:0/2201068580 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8f3801a430 con 0x7f8f481028e0 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.428+0000 7f8f4e10d700 1 -- 172.21.15.59:0/2201068580 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f8f4810cd90 con 0x7f8f481028e0 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.433+0000 7f8f2ffff700 1 -- 172.21.15.59:0/2201068580 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8f3801a630 con 0x7f8f481028e0 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.434+0000 7f8f4e10d700 1 -- 172.21.15.59:0/2201068580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 msgr2=0x7f8f48113400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.434+0000 7f8f4e10d700 1 --2- 172.21.15.59:0/2201068580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48113400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8f38011d20 tx=0x7f8f38009250 comp rx=0 tx=0).stop 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.434+0000 7f8f4e10d700 1 -- 172.21.15.59:0/2201068580 shutdown_connections 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.434+0000 7f8f4e10d700 1 --2- 172.21.15.59:0/2201068580 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8f481028e0 0x7f8f48113400 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.434+0000 7f8f4e10d700 1 -- 172.21.15.59:0/2201068580 >> 172.21.15.59:0/2201068580 conn(0x7f8f480fc0b0 msgr2=0x7f8f480fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.434+0000 7f8f4e10d700 1 -- 172.21.15.59:0/2201068580 shutdown_connections 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:02.434+0000 7f8f4e10d700 1 -- 172.21.15.59:0/2201068580 wait complete. 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:Wrote config to /etc/ceph/ceph.conf 2023-12-19T18:39:02.979 INFO:teuthology.orchestra.run.smithi059.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-19T18:39:02.980 INFO:teuthology.orchestra.run.smithi059.stdout:Creating mgr... 2023-12-19T18:39:02.980 INFO:teuthology.orchestra.run.smithi059.stdout:Verifying port 9283 ... 2023-12-19T18:39:03.151 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mgr.smithi059.ccbnob 2023-12-19T18:39:03.151 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mgr.smithi059.ccbnob.service: Unit ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mgr.smithi059.ccbnob.service not loaded. 2023-12-19T18:39:03.302 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bf854274-9e9d-11ee-95a5-87774f69a715.target.wants/ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mgr.smithi059.ccbnob.service → /etc/systemd/system/ceph-bf854274-9e9d-11ee-95a5-87774f69a715@.service. 2023-12-19T18:39:03.606 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:03 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2201068580' entity='client.admin' 2023-12-19T18:39:03.784 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-19T18:39:03.784 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2023-12-19T18:39:03.792 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-19T18:39:03.792 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2023-12-19T18:39:03.792 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2023-12-19T18:39:03.792 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to enable service . firewalld.service is not available 2023-12-19T18:39:03.801 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-19T18:39:03.801 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2023-12-19T18:39:03.809 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-19T18:39:03.809 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2023-12-19T18:39:03.809 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2023-12-19T18:39:03.809 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-19T18:39:03.809 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr to start... 2023-12-19T18:39:03.809 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr... 2023-12-19T18:39:04.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:04 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/957904684' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-19T18:39:04.812 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "bf854274-9e9d-11ee-95a5-87774f69a715", 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-19T18:39:04.825 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:04.826 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-19T18:39:04.827 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2023-12-19T18:39:04.828 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2023-12-19T18:38:57.167597+0000", 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.174+0000 7fda3ec0a700 1 Processor -- start 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.174+0000 7fda3ec0a700 1 -- start start 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.174+0000 7fda3ec0a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38104ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.174+0000 7fda3ec0a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda38105410 con 0x7fda38076c20 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.174+0000 7fda3c9a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38104ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:04.829 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.175+0000 7fda3c9a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38104ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49546/0 (socket says 172.21.15.59:49546) 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.175+0000 7fda3c9a6700 1 -- 172.21.15.59:0/1471998613 learned_addr learned my addr 172.21.15.59:0/1471998613 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.175+0000 7fda3c9a6700 1 -- 172.21.15.59:0/1471998613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda38105550 con 0x7fda38076c20 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.175+0000 7fda3c9a6700 1 --2- 172.21.15.59:0/1471998613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38104ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0098e0 tx=0x7fda2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85cc12ea1ae636c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.175+0000 7fda377fe700 1 -- 172.21.15.59:0/1471998613 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda2c00b860 con 0x7fda38076c20 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.175+0000 7fda377fe700 1 -- 172.21.15.59:0/1471998613 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fda2c00be30 con 0x7fda38076c20 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.175+0000 7fda377fe700 1 -- 172.21.15.59:0/1471998613 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda2c010380 con 0x7fda38076c20 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.176+0000 7fda3ec0a700 1 -- 172.21.15.59:0/1471998613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 msgr2=0x7fda38104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.176+0000 7fda3ec0a700 1 --2- 172.21.15.59:0/1471998613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38104ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda2c0098e0 tx=0x7fda2c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.176+0000 7fda3ec0a700 1 -- 172.21.15.59:0/1471998613 shutdown_connections 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.176+0000 7fda3ec0a700 1 --2- 172.21.15.59:0/1471998613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38104ed0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.176+0000 7fda3ec0a700 1 -- 172.21.15.59:0/1471998613 >> 172.21.15.59:0/1471998613 conn(0x7fda380fc980 msgr2=0x7fda380fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.176+0000 7fda3ec0a700 1 -- 172.21.15.59:0/1471998613 shutdown_connections 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.176+0000 7fda3ec0a700 1 -- 172.21.15.59:0/1471998613 wait complete. 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.177+0000 7fda3ec0a700 1 Processor -- start 2023-12-19T18:39:04.830 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.177+0000 7fda3ec0a700 1 -- start start 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.177+0000 7fda3ec0a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.177+0000 7fda3ec0a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda38100dd0 con 0x7fda38076c20 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.177+0000 7fda3c9a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.178+0000 7fda3c9a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38102950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49548/0 (socket says 172.21.15.59:49548) 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.178+0000 7fda3c9a6700 1 -- 172.21.15.59:0/957904684 learned_addr learned my addr 172.21.15.59:0/957904684 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.178+0000 7fda3c9a6700 1 -- 172.21.15.59:0/957904684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c0094f0 con 0x7fda38076c20 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.178+0000 7fda3c9a6700 1 --2- 172.21.15.59:0/957904684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38102950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fda2c006d40 tx=0x7fda2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.178+0000 7fda357fa700 1 -- 172.21.15.59:0/957904684 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda2c0109b0 con 0x7fda38076c20 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.178+0000 7fda357fa700 1 -- 172.21.15.59:0/957904684 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fda2c024440 con 0x7fda38076c20 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.178+0000 7fda357fa700 1 -- 172.21.15.59:0/957904684 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda2c01a5f0 con 0x7fda38076c20 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.178+0000 7fda3ec0a700 1 -- 172.21.15.59:0/957904684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda38100fd0 con 0x7fda38076c20 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.178+0000 7fda3ec0a700 1 -- 172.21.15.59:0/957904684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda381013e0 con 0x7fda38076c20 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.179+0000 7fda357fa700 1 -- 172.21.15.59:0/957904684 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fda2c01aaf0 con 0x7fda38076c20 2023-12-19T18:39:04.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.179+0000 7fda357fa700 1 -- 172.21.15.59:0/957904684 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fda2c02da40 con 0x7fda38076c20 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.179+0000 7fda3ec0a700 1 -- 172.21.15.59:0/957904684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3804bb80 con 0x7fda38076c20 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.181+0000 7fda357fa700 1 -- 172.21.15.59:0/957904684 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fda2c006d70 con 0x7fda38076c20 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.256+0000 7fda3ec0a700 1 -- 172.21.15.59:0/957904684 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fda381063f0 con 0x7fda38076c20 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.257+0000 7fda357fa700 1 -- 172.21.15.59:0/957904684 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fda2c035090 con 0x7fda38076c20 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.258+0000 7fda3ec0a700 1 -- 172.21.15.59:0/957904684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 msgr2=0x7fda38102950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.258+0000 7fda3ec0a700 1 --2- 172.21.15.59:0/957904684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38102950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fda2c006d40 tx=0x7fda2c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.258+0000 7fda3ec0a700 1 -- 172.21.15.59:0/957904684 shutdown_connections 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.258+0000 7fda3ec0a700 1 --2- 172.21.15.59:0/957904684 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fda38076c20 0x7fda38102950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.258+0000 7fda3ec0a700 1 -- 172.21.15.59:0/957904684 >> 172.21.15.59:0/957904684 conn(0x7fda380fc980 msgr2=0x7fda380fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.258+0000 7fda3ec0a700 1 -- 172.21.15.59:0/957904684 shutdown_connections 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:04.258+0000 7fda3ec0a700 1 -- 172.21.15.59:0/957904684 wait complete. 2023-12-19T18:39:04.832 INFO:teuthology.orchestra.run.smithi059.stdout:mgr not available, waiting (1/15)... 2023-12-19T18:39:07.778 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:39:07.778 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2023-12-19T18:39:07.778 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "bf854274-9e9d-11ee-95a5-87774f69a715", 2023-12-19T18:39:07.778 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2023-12-19T18:39:07.778 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-19T18:39:07.779 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-19T18:39:07.780 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-19T18:39:07.781 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2023-12-19T18:38:57.167597+0000", 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.194+0000 7fd4654fa700 1 Processor -- start 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.194+0000 7fd4654fa700 1 -- start start 2023-12-19T18:39:07.782 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.195+0000 7fd4654fa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4600760e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:07.783 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.195+0000 7fd4654fa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd460077fe0 con 0x7fd460077bd0 2023-12-19T18:39:07.783 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.195+0000 7fd45effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4600760e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:07.783 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.195+0000 7fd45effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4600760e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49550/0 (socket says 172.21.15.59:49550) 2023-12-19T18:39:07.783 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.195+0000 7fd45effd700 1 -- 172.21.15.59:0/2212846947 learned_addr learned my addr 172.21.15.59:0/2212846947 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:07.783 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.195+0000 7fd45effd700 1 -- 172.21.15.59:0/2212846947 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd460076620 con 0x7fd460077bd0 2023-12-19T18:39:07.783 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.195+0000 7fd45effd700 1 --2- 172.21.15.59:0/2212846947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4600760e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd4500098e0 tx=0x7fd450009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d03723426aed25 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:07.783 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.196+0000 7fd45dffb700 1 -- 172.21.15.59:0/2212846947 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd45000b860 con 0x7fd460077bd0 2023-12-19T18:39:07.783 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.196+0000 7fd45dffb700 1 -- 172.21.15.59:0/2212846947 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd45000be30 con 0x7fd460077bd0 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.196+0000 7fd45dffb700 1 -- 172.21.15.59:0/2212846947 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd450010380 con 0x7fd460077bd0 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.197+0000 7fd4654fa700 1 -- 172.21.15.59:0/2212846947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 msgr2=0x7fd4600760e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.197+0000 7fd4654fa700 1 --2- 172.21.15.59:0/2212846947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4600760e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd4500098e0 tx=0x7fd450009250 comp rx=0 tx=0).stop 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.197+0000 7fd4654fa700 1 -- 172.21.15.59:0/2212846947 shutdown_connections 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.197+0000 7fd4654fa700 1 --2- 172.21.15.59:0/2212846947 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4600760e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.197+0000 7fd4654fa700 1 -- 172.21.15.59:0/2212846947 >> 172.21.15.59:0/2212846947 conn(0x7fd4600fda50 msgr2=0x7fd4600ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.197+0000 7fd4654fa700 1 -- 172.21.15.59:0/2212846947 shutdown_connections 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.197+0000 7fd4654fa700 1 -- 172.21.15.59:0/2212846947 wait complete. 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.197+0000 7fd4654fa700 1 Processor -- start 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.197+0000 7fd4654fa700 1 -- start start 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd4654fa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4601133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd4654fa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd460113900 con 0x7fd460077bd0 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd45effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4601133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd45effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4601133c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49556/0 (socket says 172.21.15.59:49556) 2023-12-19T18:39:07.784 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd45effd700 1 -- 172.21.15.59:0/1670218531 learned_addr learned my addr 172.21.15.59:0/1670218531 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd45effd700 1 -- 172.21.15.59:0/1670218531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4500094f0 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd45effd700 1 --2- 172.21.15.59:0/1670218531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4601133c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd450006fd0 tx=0x7fd450009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd4577fe700 1 -- 172.21.15.59:0/1670218531 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd450010750 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd4654fa700 1 -- 172.21.15.59:0/1670218531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd460113b00 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd4577fe700 1 -- 172.21.15.59:0/1670218531 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd450010d20 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd4577fe700 1 -- 172.21.15.59:0/1670218531 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd45001ada0 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.198+0000 7fd4654fa700 1 -- 172.21.15.59:0/1670218531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd460113f10 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.199+0000 7fd4577fe700 1 -- 172.21.15.59:0/1670218531 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd450024680 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.199+0000 7fd4577fe700 1 -- 172.21.15.59:0/1670218531 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd450019880 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.199+0000 7fd4654fa700 1 -- 172.21.15.59:0/1670218531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46004bb80 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.202+0000 7fd4577fe700 1 -- 172.21.15.59:0/1670218531 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd450029060 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.276+0000 7fd4654fa700 1 -- 172.21.15.59:0/1670218531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd460108540 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.276+0000 7fd4577fe700 1 -- 172.21.15.59:0/1670218531 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fd460108540 con 0x7fd460077bd0 2023-12-19T18:39:07.785 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.277+0000 7fd4654fa700 1 -- 172.21.15.59:0/1670218531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 msgr2=0x7fd4601133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:07.786 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.277+0000 7fd4654fa700 1 --2- 172.21.15.59:0/1670218531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4601133c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd450006fd0 tx=0x7fd450009250 comp rx=0 tx=0).stop 2023-12-19T18:39:07.786 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.277+0000 7fd4654fa700 1 -- 172.21.15.59:0/1670218531 shutdown_connections 2023-12-19T18:39:07.786 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.277+0000 7fd4654fa700 1 --2- 172.21.15.59:0/1670218531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd460077bd0 0x7fd4601133c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:07.786 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.277+0000 7fd4654fa700 1 -- 172.21.15.59:0/1670218531 >> 172.21.15.59:0/1670218531 conn(0x7fd4600fda50 msgr2=0x7fd4600ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:07.786 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.277+0000 7fd4654fa700 1 -- 172.21.15.59:0/1670218531 shutdown_connections 2023-12-19T18:39:07.786 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:07.277+0000 7fd4654fa700 1 -- 172.21.15.59:0/1670218531 wait complete. 2023-12-19T18:39:07.786 INFO:teuthology.orchestra.run.smithi059.stdout:mgr not available, waiting (2/15)... 2023-12-19T18:39:08.141 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:07 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1670218531' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-19T18:39:10.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: Activating manager daemon smithi059.ccbnob 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: mgrmap e2: smithi059.ccbnob(active, starting, since 0.00309654s) 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr metadata", "who": "smithi059.ccbnob", "id": "smithi059.ccbnob"}]: dispatch 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: Manager daemon smithi059.ccbnob is now available 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/mirror_snapshot_schedule"}]: dispatch 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/trash_purge_schedule"}]: dispatch 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:10.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:10 smithi059 ceph-mon[80661]: from='mgr.14100 172.21.15.59:0/3499440651' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:12.821 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:12 smithi059 ceph-mon[80661]: mgrmap e3: smithi059.ccbnob(active, since 1.00743s) 2023-12-19T18:39:13.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:13 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3572331508' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-19T18:39:14.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:14 smithi059 ceph-mon[80661]: mgrmap e4: smithi059.ccbnob(active, since 3s) 2023-12-19T18:39:18.630 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:39:18.630 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2023-12-19T18:39:18.630 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsid": "bf854274-9e9d-11ee-95a5-87774f69a715", 2023-12-19T18:39:18.630 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "health": { 2023-12-19T18:39:18.630 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 0 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "smithi059" 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-19T18:39:18.631 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:18.632 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-19T18:39:18.632 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:18.632 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-19T18:39:18.632 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-19T18:39:18.632 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-19T18:39:18.641 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-19T18:39:18.642 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-19T18:39:18.642 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-19T18:39:18.642 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:18.642 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-19T18:39:18.642 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-19T18:39:18.642 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-19T18:39:18.643 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-19T18:39:18.643 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-19T18:39:18.643 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-19T18:39:18.643 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-19T18:39:18.643 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-19T18:39:18.643 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-19T18:39:18.643 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:18.643 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-19T18:39:18.644 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:18.644 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-19T18:39:18.644 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-19T18:39:18.644 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:18.644 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-19T18:39:18.644 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-19T18:39:18.644 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-19T18:39:18.645 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-19T18:39:18.645 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-19T18:39:18.645 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-19T18:39:18.645 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "restful" 2023-12-19T18:39:18.645 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ], 2023-12-19T18:39:18.645 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T18:39:18.645 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:18.646 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-19T18:39:18.646 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T18:39:18.646 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "modified": "2023-12-19T18:38:57.167597+0000", 2023-12-19T18:39:18.646 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T18:39:18.646 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout }, 2023-12-19T18:39:18.646 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-19T18:39:18.646 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2023-12-19T18:39:18.647 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.821+0000 7effa93af700 1 Processor -- start 2023-12-19T18:39:18.647 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.821+0000 7effa93af700 1 -- start start 2023-12-19T18:39:18.647 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.821+0000 7effa93af700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:18.647 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.821+0000 7effa93af700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa4105420 con 0x7effa4076c20 2023-12-19T18:39:18.647 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.821+0000 7effa2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:18.647 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.821+0000 7effa2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48260/0 (socket says 172.21.15.59:48260) 2023-12-19T18:39:18.648 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.821+0000 7effa2ffd700 1 -- 172.21.15.59:0/4209346290 learned_addr learned my addr 172.21.15.59:0/4209346290 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:18.648 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.822+0000 7effa2ffd700 1 -- 172.21.15.59:0/4209346290 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa4105560 con 0x7effa4076c20 2023-12-19T18:39:18.648 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.822+0000 7effa2ffd700 1 --2- 172.21.15.59:0/4209346290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4104ee0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7eff940098e0 tx=0x7eff94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59fae70766c10190 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:18.648 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.822+0000 7effa1ffb700 1 -- 172.21.15.59:0/4209346290 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff9400b860 con 0x7effa4076c20 2023-12-19T18:39:18.648 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.822+0000 7effa1ffb700 1 -- 172.21.15.59:0/4209346290 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7eff9400be30 con 0x7effa4076c20 2023-12-19T18:39:18.648 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.822+0000 7effa1ffb700 1 -- 172.21.15.59:0/4209346290 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff94010380 con 0x7effa4076c20 2023-12-19T18:39:18.648 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.823+0000 7effa93af700 1 -- 172.21.15.59:0/4209346290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 msgr2=0x7effa4104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:18.649 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.823+0000 7effa93af700 1 --2- 172.21.15.59:0/4209346290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4104ee0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7eff940098e0 tx=0x7eff94009250 comp rx=0 tx=0).stop 2023-12-19T18:39:18.649 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.823+0000 7effa93af700 1 -- 172.21.15.59:0/4209346290 shutdown_connections 2023-12-19T18:39:18.649 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.823+0000 7effa93af700 1 --2- 172.21.15.59:0/4209346290 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4104ee0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:18.649 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.823+0000 7effa93af700 1 -- 172.21.15.59:0/4209346290 >> 172.21.15.59:0/4209346290 conn(0x7effa40fc950 msgr2=0x7effa40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:18.649 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.823+0000 7effa93af700 1 -- 172.21.15.59:0/4209346290 shutdown_connections 2023-12-19T18:39:18.649 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.823+0000 7effa93af700 1 -- 172.21.15.59:0/4209346290 wait complete. 2023-12-19T18:39:18.649 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.824+0000 7effa93af700 1 Processor -- start 2023-12-19T18:39:18.650 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.824+0000 7effa93af700 1 -- start start 2023-12-19T18:39:18.650 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.824+0000 7effa93af700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4111160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:18.650 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.824+0000 7effa93af700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa41116a0 con 0x7effa4076c20 2023-12-19T18:39:18.650 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.824+0000 7effa2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:18.650 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.824+0000 7effa2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4111160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48274/0 (socket says 172.21.15.59:48274) 2023-12-19T18:39:18.650 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.824+0000 7effa2ffd700 1 -- 172.21.15.59:0/3572331508 learned_addr learned my addr 172.21.15.59:0/3572331508 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:18.651 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.824+0000 7effa2ffd700 1 -- 172.21.15.59:0/3572331508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff940094f0 con 0x7effa4076c20 2023-12-19T18:39:18.651 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.824+0000 7effa2ffd700 1 --2- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4111160 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff94006b00 tx=0x7eff94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:18.651 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.825+0000 7eff8b7fe700 1 -- 172.21.15.59:0/3572331508 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff94010a80 con 0x7effa4076c20 2023-12-19T18:39:18.651 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.825+0000 7eff8b7fe700 1 -- 172.21.15.59:0/3572331508 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7eff9401a430 con 0x7effa4076c20 2023-12-19T18:39:18.651 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.825+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa41118a0 con 0x7effa4076c20 2023-12-19T18:39:18.651 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.825+0000 7eff8b7fe700 1 -- 172.21.15.59:0/3572331508 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff940245b0 con 0x7effa4076c20 2023-12-19T18:39:18.651 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.825+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa4111cb0 con 0x7effa4076c20 2023-12-19T18:39:18.652 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.825+0000 7eff8b7fe700 1 -- 172.21.15.59:0/3572331508 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 3) v1 ==== 43164+0+0 (secure 0 0 0) 0x7eff940247e0 con 0x7effa4076c20 2023-12-19T18:39:18.652 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.826+0000 7eff8b7fe700 1 --2- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff8c0364c0 0x7eff8c038970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:18.652 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.826+0000 7eff8b7fe700 1 -- 172.21.15.59:0/3572331508 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7eff9404a880 con 0x7effa4076c20 2023-12-19T18:39:18.652 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.826+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa404bb80 con 0x7effa4076c20 2023-12-19T18:39:18.652 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.826+0000 7effa27fc700 1 --2- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff8c0364c0 0x7eff8c038970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:18.652 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.826+0000 7effa27fc700 1 --2- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff8c0364c0 0x7eff8c038970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff98009990 tx=0x7eff98006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:18.652 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:12.831+0000 7eff8b7fe700 1 -- 172.21.15.59:0/3572331508 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7eff94024ae0 con 0x7effa4076c20 2023-12-19T18:39:18.653 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.106+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7effa410aea0 con 0x7effa4076c20 2023-12-19T18:39:18.653 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.107+0000 7eff8b7fe700 1 -- 172.21.15.59:0/3572331508 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7eff9402cb30 con 0x7effa4076c20 2023-12-19T18:39:18.653 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff8c0364c0 msgr2=0x7eff8c038970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:18.653 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 --2- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff8c0364c0 0x7eff8c038970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff98009990 tx=0x7eff98006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:18.653 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 msgr2=0x7effa4111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:18.653 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 --2- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4111160 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff94006b00 tx=0x7eff94009250 comp rx=0 tx=0).stop 2023-12-19T18:39:18.653 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 shutdown_connections 2023-12-19T18:39:18.654 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 --2- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff8c0364c0 0x7eff8c038970 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:18.654 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 --2- 172.21.15.59:0/3572331508 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effa4076c20 0x7effa4111160 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:18.654 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 >> 172.21.15.59:0/3572331508 conn(0x7effa40fc950 msgr2=0x7effa40fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:18.654 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 shutdown_connections 2023-12-19T18:39:18.654 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:13.110+0000 7effa93af700 1 -- 172.21.15.59:0/3572331508 wait complete. 2023-12-19T18:39:18.654 INFO:teuthology.orchestra.run.smithi059.stdout:mgr is available 2023-12-19T18:39:19.525 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:19 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2198863892' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [global] 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout fsid = bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [mgr] 2023-12-19T18:39:19.831 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout [osd] 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.037+0000 7f5d0579f700 1 Processor -- start 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.038+0000 7f5d0579f700 1 -- start start 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.038+0000 7f5d0579f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00101430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.038+0000 7f5d0579f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d00101970 con 0x7f5d00101020 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.038+0000 7f5cfffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.038+0000 7f5cfffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00101430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48286/0 (socket says 172.21.15.59:48286) 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.038+0000 7f5cfffff700 1 -- 172.21.15.59:0/728463493 learned_addr learned my addr 172.21.15.59:0/728463493 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:19.832 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.038+0000 7f5cfffff700 1 -- 172.21.15.59:0/728463493 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d00101ab0 con 0x7f5d00101020 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.038+0000 7f5cfffff700 1 --2- 172.21.15.59:0/728463493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00101430 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00098e0 tx=0x7f5cf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8837c789eb7b707 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5cfeffd700 1 -- 172.21.15.59:0/728463493 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf000b860 con 0x7f5d00101020 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5cfeffd700 1 -- 172.21.15.59:0/728463493 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5cf000be30 con 0x7f5d00101020 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5cfeffd700 1 -- 172.21.15.59:0/728463493 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf0010380 con 0x7f5d00101020 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5d0579f700 1 -- 172.21.15.59:0/728463493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 msgr2=0x7f5d00101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5d0579f700 1 --2- 172.21.15.59:0/728463493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00101430 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00098e0 tx=0x7f5cf0009250 comp rx=0 tx=0).stop 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5d0579f700 1 -- 172.21.15.59:0/728463493 shutdown_connections 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5d0579f700 1 --2- 172.21.15.59:0/728463493 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00101430 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5d0579f700 1 -- 172.21.15.59:0/728463493 >> 172.21.15.59:0/728463493 conn(0x7f5d000fc760 msgr2=0x7f5d000febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5d0579f700 1 -- 172.21.15.59:0/728463493 shutdown_connections 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.039+0000 7f5d0579f700 1 -- 172.21.15.59:0/728463493 wait complete. 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.040+0000 7f5d0579f700 1 Processor -- start 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.040+0000 7f5d0579f700 1 -- start start 2023-12-19T18:39:19.833 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.040+0000 7f5d0579f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00113c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.040+0000 7f5d0579f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d00070f90 con 0x7f5d00101020 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5cfffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00113c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5cfffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00113c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48296/0 (socket says 172.21.15.59:48296) 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5cfffff700 1 -- 172.21.15.59:0/2198863892 learned_addr learned my addr 172.21.15.59:0/2198863892 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5cfffff700 1 -- 172.21.15.59:0/2198863892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf00094f0 con 0x7f5d00101020 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5cfffff700 1 --2- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00113c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0006b00 tx=0x7f5cf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5cfcff9700 1 -- 172.21.15.59:0/2198863892 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf00109b0 con 0x7f5d00101020 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5cfcff9700 1 -- 172.21.15.59:0/2198863892 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5cf0024440 con 0x7f5d00101020 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d00071190 con 0x7f5d00101020 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5cfcff9700 1 -- 172.21.15.59:0/2198863892 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf001a5f0 con 0x7f5d00101020 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.041+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d00114390 con 0x7f5d00101020 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.042+0000 7f5cfcff9700 1 -- 172.21.15.59:0/2198863892 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f5cf001a860 con 0x7f5d00101020 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.042+0000 7f5cfcff9700 1 --2- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f5ce80365f0 0x7f5ce8038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:19.834 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.043+0000 7f5cfcff9700 1 -- 172.21.15.59:0/2198863892 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5cf004a540 con 0x7f5d00101020 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.043+0000 7f5cff7fe700 1 --2- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f5ce80365f0 0x7f5ce8038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.043+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d00109ea0 con 0x7f5d00101020 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.043+0000 7f5cff7fe700 1 --2- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f5ce80365f0 0x7f5ce8038aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009990 tx=0x7f5cf4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.048+0000 7f5cfcff9700 1 -- 172.21.15.59:0/2198863892 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5cf001ab10 con 0x7f5d00101020 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.242+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f5d00102e30 con 0x7f5d00101020 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.243+0000 7f5cfcff9700 1 -- 172.21.15.59:0/2198863892 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f5cf0018b40 con 0x7f5d00101020 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f5ce80365f0 msgr2=0x7f5ce8038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 --2- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f5ce80365f0 0x7f5ce8038aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009990 tx=0x7f5cf4006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 msgr2=0x7f5d00113c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 --2- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00113c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0006b00 tx=0x7f5cf0009250 comp rx=0 tx=0).stop 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 shutdown_connections 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 --2- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f5ce80365f0 0x7f5ce8038aa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 --2- 172.21.15.59:0/2198863892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d00101020 0x7f5d00113c90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 >> 172.21.15.59:0/2198863892 conn(0x7f5d000fc760 msgr2=0x7f5d000febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:19.835 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 shutdown_connections 2023-12-19T18:39:19.836 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:19.245+0000 7f5d0579f700 1 -- 172.21.15.59:0/2198863892 wait complete. 2023-12-19T18:39:19.836 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling cephadm module... 2023-12-19T18:39:20.911 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:20 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3936597483' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-19T18:39:21.219 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.244+0000 7eff41ed0700 1 Processor -- start 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.244+0000 7eff41ed0700 1 -- start start 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.244+0000 7eff41ed0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.244+0000 7eff41ed0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c103250 con 0x7eff3c102900 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.244+0000 7eff3b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.244+0000 7eff3b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48302/0 (socket says 172.21.15.59:48302) 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.244+0000 7eff3b7fe700 1 -- 172.21.15.59:0/1406806079 learned_addr learned my addr 172.21.15.59:0/1406806079 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.244+0000 7eff3b7fe700 1 -- 172.21.15.59:0/1406806079 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c103390 con 0x7eff3c102900 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.245+0000 7eff3b7fe700 1 --2- 172.21.15.59:0/1406806079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c102d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0098e0 tx=0x7eff2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7323dd84a40b8b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.245+0000 7eff3a7fc700 1 -- 172.21.15.59:0/1406806079 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c00b860 con 0x7eff3c102900 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.245+0000 7eff3a7fc700 1 -- 172.21.15.59:0/1406806079 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7eff2c00be30 con 0x7eff3c102900 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.245+0000 7eff3a7fc700 1 -- 172.21.15.59:0/1406806079 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c010380 con 0x7eff3c102900 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.245+0000 7eff41ed0700 1 -- 172.21.15.59:0/1406806079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 msgr2=0x7eff3c102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:21.220 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.245+0000 7eff41ed0700 1 --2- 172.21.15.59:0/1406806079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c102d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0098e0 tx=0x7eff2c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.245+0000 7eff41ed0700 1 -- 172.21.15.59:0/1406806079 shutdown_connections 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.245+0000 7eff41ed0700 1 --2- 172.21.15.59:0/1406806079 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c102d10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.245+0000 7eff41ed0700 1 -- 172.21.15.59:0/1406806079 >> 172.21.15.59:0/1406806079 conn(0x7eff3c0fc0b0 msgr2=0x7eff3c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.246+0000 7eff41ed0700 1 -- 172.21.15.59:0/1406806079 shutdown_connections 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.246+0000 7eff41ed0700 1 -- 172.21.15.59:0/1406806079 wait complete. 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.246+0000 7eff41ed0700 1 Processor -- start 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.246+0000 7eff41ed0700 1 -- start start 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff41ed0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c113360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff41ed0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c1138a0 con 0x7eff3c102900 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff3b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c113360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff3b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c113360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48316/0 (socket says 172.21.15.59:48316) 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff3b7fe700 1 -- 172.21.15.59:0/3936597483 learned_addr learned my addr 172.21.15.59:0/3936597483 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff3b7fe700 1 -- 172.21.15.59:0/3936597483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c0094f0 con 0x7eff3c102900 2023-12-19T18:39:21.221 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff3b7fe700 1 --2- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c113360 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7eff2c006b00 tx=0x7eff2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff23fff700 1 -- 172.21.15.59:0/3936597483 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c0109b0 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff23fff700 1 -- 172.21.15.59:0/3936597483 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7eff2c024440 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.247+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3c113aa0 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.248+0000 7eff23fff700 1 -- 172.21.15.59:0/3936597483 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c01a5f0 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.248+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3c113eb0 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.248+0000 7eff23fff700 1 -- 172.21.15.59:0/3936597483 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7eff2c01a8b0 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.249+0000 7eff23fff700 1 --2- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff240365a0 0x7eff24038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.249+0000 7eff23fff700 1 -- 172.21.15.59:0/3936597483 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7eff2c04a570 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.249+0000 7eff3affd700 1 --2- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff240365a0 0x7eff24038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.249+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3c04bb80 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.249+0000 7eff3affd700 1 --2- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff240365a0 0x7eff24038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7eff30009990 tx=0x7eff30006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.258+0000 7eff23fff700 1 -- 172.21.15.59:0/3936597483 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7eff2c022340 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.507+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7eff3c10ce20 con 0x7eff3c102900 2023-12-19T18:39:21.222 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.559+0000 7eff23fff700 1 -- 172.21.15.59:0/3936597483 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7eff2c00fdc0 con 0x7eff3c102900 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.560+0000 7eff23fff700 1 -- 172.21.15.59:0/3936597483 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7eff2c00f2e0 con 0x7eff3c102900 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.563+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff240365a0 msgr2=0x7eff24038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.563+0000 7eff41ed0700 1 --2- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff240365a0 0x7eff24038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7eff30009990 tx=0x7eff30006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.563+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 msgr2=0x7eff3c113360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.563+0000 7eff41ed0700 1 --2- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c113360 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7eff2c006b00 tx=0x7eff2c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.563+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 shutdown_connections 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.563+0000 7eff41ed0700 1 --2- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7eff240365a0 0x7eff24038a50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.563+0000 7eff41ed0700 1 --2- 172.21.15.59:0/3936597483 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff3c102900 0x7eff3c113360 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.563+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 >> 172.21.15.59:0/3936597483 conn(0x7eff3c0fc0b0 msgr2=0x7eff3c0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.564+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 shutdown_connections 2023-12-19T18:39:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:20.564+0000 7eff41ed0700 1 -- 172.21.15.59:0/3936597483 wait complete. 2023-12-19T18:39:21.828 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:21 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3936597483' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-19T18:39:21.829 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:21 smithi059 ceph-mon[80661]: mgrmap e5: smithi059.ccbnob(active, since 9s) 2023-12-19T18:39:22.434 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "active_name": "smithi059.ccbnob", 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.623+0000 7f2c951a5700 1 Processor -- start 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.623+0000 7f2c951a5700 1 -- start start 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.623+0000 7f2c951a5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c90104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.623+0000 7f2c951a5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c90105440 con 0x7f2c90070fa0 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.624+0000 7f2c8ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c90104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.624+0000 7f2c8ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c90104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37960/0 (socket says 172.21.15.59:37960) 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.624+0000 7f2c8ed9d700 1 -- 172.21.15.59:0/4231895100 learned_addr learned my addr 172.21.15.59:0/4231895100 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:22.435 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.624+0000 7f2c8ed9d700 1 -- 172.21.15.59:0/4231895100 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c90105580 con 0x7f2c90070fa0 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.624+0000 7f2c8ed9d700 1 --2- 172.21.15.59:0/4231895100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c90104f00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2c840098e0 tx=0x7f2c84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8be92742c3197a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.624+0000 7f2c8dd9b700 1 -- 172.21.15.59:0/4231895100 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8400b860 con 0x7f2c90070fa0 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.624+0000 7f2c8dd9b700 1 -- 172.21.15.59:0/4231895100 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2c8400be30 con 0x7f2c90070fa0 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.624+0000 7f2c8dd9b700 1 -- 172.21.15.59:0/4231895100 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c84010380 con 0x7f2c90070fa0 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.625+0000 7f2c951a5700 1 -- 172.21.15.59:0/4231895100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 msgr2=0x7f2c90104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.625+0000 7f2c951a5700 1 --2- 172.21.15.59:0/4231895100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c90104f00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2c840098e0 tx=0x7f2c84009250 comp rx=0 tx=0).stop 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.625+0000 7f2c951a5700 1 -- 172.21.15.59:0/4231895100 shutdown_connections 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.625+0000 7f2c951a5700 1 --2- 172.21.15.59:0/4231895100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c90104f00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.625+0000 7f2c951a5700 1 -- 172.21.15.59:0/4231895100 >> 172.21.15.59:0/4231895100 conn(0x7f2c900fc970 msgr2=0x7f2c900fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.625+0000 7f2c951a5700 1 -- 172.21.15.59:0/4231895100 shutdown_connections 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.625+0000 7f2c951a5700 1 -- 172.21.15.59:0/4231895100 wait complete. 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.626+0000 7f2c951a5700 1 Processor -- start 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.626+0000 7f2c951a5700 1 -- start start 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.626+0000 7f2c951a5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c901111a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.626+0000 7f2c951a5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c901116e0 con 0x7f2c90070fa0 2023-12-19T18:39:22.436 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.626+0000 7f2c8ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c901111a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.626+0000 7f2c8ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c901111a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37972/0 (socket says 172.21.15.59:37972) 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.626+0000 7f2c8ed9d700 1 -- 172.21.15.59:0/2570543351 learned_addr learned my addr 172.21.15.59:0/2570543351 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.626+0000 7f2c8ed9d700 1 -- 172.21.15.59:0/2570543351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c840094f0 con 0x7f2c90070fa0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.627+0000 7f2c8ed9d700 1 --2- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c901111a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c84006d40 tx=0x7f2c84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.627+0000 7f2c777fe700 1 -- 172.21.15.59:0/2570543351 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c840109b0 con 0x7f2c90070fa0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.627+0000 7f2c777fe700 1 -- 172.21.15.59:0/2570543351 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2c84024440 con 0x7f2c90070fa0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.627+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c901118e0 con 0x7f2c90070fa0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.627+0000 7f2c777fe700 1 -- 172.21.15.59:0/2570543351 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c8401a5f0 con 0x7f2c90070fa0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.627+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c90111cf0 con 0x7f2c90070fa0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.628+0000 7f2c777fe700 1 -- 172.21.15.59:0/2570543351 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f2c84010b10 con 0x7f2c90070fa0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.628+0000 7f2c777fe700 1 --2- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f2c78036640 0x7f2c78038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.628+0000 7f2c777fe700 1 -- 172.21.15.59:0/2570543351 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2c8404a420 con 0x7f2c90070fa0 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.628+0000 7f2c8e59c700 1 -- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f2c78036640 msgr2=0x7f2c78038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1973265269 2023-12-19T18:39:22.437 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.628+0000 7f2c8e59c700 1 --2- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f2c78036640 0x7f2c78038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.628+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9004bb80 con 0x7f2c90070fa0 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.633+0000 7f2c777fe700 1 -- 172.21.15.59:0/2570543351 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2c8402d3f0 con 0x7f2c90070fa0 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.829+0000 7f2c8e59c700 1 -- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f2c78036640 msgr2=0x7f2c78038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1973265269 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.829+0000 7f2c8e59c700 1 --2- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f2c78036640 0x7f2c78038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.887+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2c9010acb0 con 0x7f2c90070fa0 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.887+0000 7f2c777fe700 1 -- 172.21.15.59:0/2570543351 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f2c840233a0 con 0x7f2c90070fa0 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f2c78036640 msgr2=0x7f2c78038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 --2- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f2c78036640 0x7f2c78038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 msgr2=0x7f2c901111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 --2- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c901111a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2c84006d40 tx=0x7f2c84009250 comp rx=0 tx=0).stop 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 shutdown_connections 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 --2- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f2c78036640 0x7f2c78038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 --2- 172.21.15.59:0/2570543351 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c90070fa0 0x7f2c901111a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 >> 172.21.15.59:0/2570543351 conn(0x7f2c900fc970 msgr2=0x7f2c900fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:22.438 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 shutdown_connections 2023-12-19T18:39:22.439 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:21.891+0000 7f2c951a5700 1 -- 172.21.15.59:0/2570543351 wait complete. 2023-12-19T18:39:22.439 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for the mgr to restart... 2023-12-19T18:39:22.439 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr epoch 5... 2023-12-19T18:39:22.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:22 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2570543351' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-19T18:39:28.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: Active manager daemon smithi059.ccbnob restarted 2023-12-19T18:39:28.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: Activating manager daemon smithi059.ccbnob 2023-12-19T18:39:28.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: osdmap e2: 0 total, 0 up, 0 in 2023-12-19T18:39:28.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: mgrmap e6: smithi059.ccbnob(active, starting, since 0.00356081s) 2023-12-19T18:39:28.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:39:28.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr metadata", "who": "smithi059.ccbnob", "id": "smithi059.ccbnob"}]: dispatch 2023-12-19T18:39:28.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: Manager daemon smithi059.ccbnob is now available 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: Found migration_current of "None". Setting to last migration. 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/mirror_snapshot_schedule"}]: dispatch 2023-12-19T18:39:28.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:27 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/trash_purge_schedule"}]: dispatch 2023-12-19T18:39:28.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2023-12-19T18:39:28.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-12-19T18:39:28.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "initialized": true 2023-12-19T18:39:28.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2023-12-19T18:39:28.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.823+0000 7f75f9b2f700 1 Processor -- start 2023-12-19T18:39:28.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.823+0000 7f75f9b2f700 1 -- start start 2023-12-19T18:39:28.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.823+0000 7f75f9b2f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f41026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:28.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.823+0000 7f75f9b2f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4102c00 con 0x7f75f41022b0 2023-12-19T18:39:28.975 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.823+0000 7f75f37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f41026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.823+0000 7f75f37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f41026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37974/0 (socket says 172.21.15.59:37974) 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.823+0000 7f75f37fe700 1 -- 172.21.15.59:0/710513572 learned_addr learned my addr 172.21.15.59:0/710513572 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.823+0000 7f75f37fe700 1 -- 172.21.15.59:0/710513572 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4102d40 con 0x7f75f41022b0 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.824+0000 7f75f37fe700 1 --2- 172.21.15.59:0/710513572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f41026c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75e40098e0 tx=0x7f75e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fabefcd8391ccd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.824+0000 7f75f27fc700 1 -- 172.21.15.59:0/710513572 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75e400b860 con 0x7f75f41022b0 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.824+0000 7f75f27fc700 1 -- 172.21.15.59:0/710513572 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f75e400be30 con 0x7f75f41022b0 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.824+0000 7f75f27fc700 1 -- 172.21.15.59:0/710513572 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75e4010380 con 0x7f75f41022b0 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.824+0000 7f75f9b2f700 1 -- 172.21.15.59:0/710513572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 msgr2=0x7f75f41026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.824+0000 7f75f9b2f700 1 --2- 172.21.15.59:0/710513572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f41026c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f75e40098e0 tx=0x7f75e4009250 comp rx=0 tx=0).stop 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.824+0000 7f75f9b2f700 1 -- 172.21.15.59:0/710513572 shutdown_connections 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.824+0000 7f75f9b2f700 1 --2- 172.21.15.59:0/710513572 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f41026c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.824+0000 7f75f9b2f700 1 -- 172.21.15.59:0/710513572 >> 172.21.15.59:0/710513572 conn(0x7f75f40fda10 msgr2=0x7f75f40ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.825+0000 7f75f9b2f700 1 -- 172.21.15.59:0/710513572 shutdown_connections 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.825+0000 7f75f9b2f700 1 -- 172.21.15.59:0/710513572 wait complete. 2023-12-19T18:39:28.976 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.825+0000 7f75f9b2f700 1 Processor -- start 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.825+0000 7f75f9b2f700 1 -- start start 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.825+0000 7f75f9b2f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f4078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.825+0000 7f75f9b2f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4077190 con 0x7f75f41022b0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75f37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f4078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75f37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f4078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37988/0 (socket says 172.21.15.59:37988) 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75f37fe700 1 -- 172.21.15.59:0/1968231965 learned_addr learned my addr 172.21.15.59:0/1968231965 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75f37fe700 1 -- 172.21.15.59:0/1968231965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e40094f0 con 0x7f75f41022b0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75f37fe700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f4078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f75e4000c00 tx=0x7f75e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75e4010a80 con 0x7f75f41022b0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f75e401a430 con 0x7f75f41022b0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75f9b2f700 1 -- 172.21.15.59:0/1968231965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f4077390 con 0x7f75f41022b0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75f9b2f700 1 -- 172.21.15.59:0/1968231965 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f40777a0 con 0x7f75f41022b0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.826+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75e40245b0 con 0x7f75f41022b0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.827+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f75e40247a0 con 0x7f75f41022b0 2023-12-19T18:39:28.977 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.827+0000 7f75dbfff700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 0x7f75dc038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.827+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 --> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75dc039200 con 0x7f75dc036640 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.827+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f75e404ab30 con 0x7f75f41022b0 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.827+0000 7f75f2ffd700 1 -- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 msgr2=0x7f75dc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1973265269 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:22.828+0000 7f75f2ffd700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 0x7f75dc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:23.029+0000 7f75f2ffd700 1 -- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 msgr2=0x7f75dc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1973265269 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:23.029+0000 7f75f2ffd700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 0x7f75dc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:23.430+0000 7f75f2ffd700 1 -- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 msgr2=0x7f75dc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1973265269 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:23.430+0000 7f75f2ffd700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 0x7f75dc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:24.231+0000 7f75f2ffd700 1 -- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 msgr2=0x7f75dc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1973265269 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:24.231+0000 7f75f2ffd700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 0x7f75dc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:25.832+0000 7f75f2ffd700 1 -- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 msgr2=0x7f75dc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/1973265269 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:25.832+0000 7f75f2ffd700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 0x7f75dc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:27.398+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 6) v1 ==== 43048+0+0 (secure 0 0 0) 0x7f75e4024cd0 con 0x7f75f41022b0 2023-12-19T18:39:28.978 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:27.398+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 msgr2=0x7f75dc038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:27.398+0000 7f75dbfff700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 0x7f75dc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.399+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7f75e402e440 con 0x7f75f41022b0 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.399+0000 7f75dbfff700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f75dc039f60 0x7f75dc03c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.399+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75dc039200 con 0x7f75dc039f60 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.400+0000 7f75f2ffd700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f75dc039f60 0x7f75dc03c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.400+0000 7f75f2ffd700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f75dc039f60 0x7f75dc03c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75e80039d0 tx=0x7f75e8009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.401+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f75dc039200 con 0x7f75dc039f60 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.407+0000 7f75f9b2f700 1 -- 172.21.15.59:0/1968231965 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f75f405f270 con 0x7f75dc039f60 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.407+0000 7f75dbfff700 1 -- 172.21.15.59:0/1968231965 <== mgr.14118 v2:172.21.15.59:6800/2067989811 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f75f405f270 con 0x7f75dc039f60 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 -- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f75dc039f60 msgr2=0x7f75dc03c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f75dc039f60 0x7f75dc03c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75e80039d0 tx=0x7f75e8009350 comp rx=0 tx=0).stop 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 -- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 msgr2=0x7f75f4078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f4078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f75e4000c00 tx=0x7f75e4009250 comp rx=0 tx=0).stop 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 -- 172.21.15.59:0/1968231965 shutdown_connections 2023-12-19T18:39:28.979 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f75dc039f60 0x7f75dc03c340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:28.980 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75f41022b0 0x7f75f4078d10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:28.980 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 --2- 172.21.15.59:0/1968231965 >> [v2:172.21.15.59:6800/1973265269,v1:172.21.15.59:6801/1973265269] conn(0x7f75dc036640 0x7f75dc038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:28.980 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 -- 172.21.15.59:0/1968231965 >> 172.21.15.59:0/1968231965 conn(0x7f75f40fda10 msgr2=0x7f75f40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:28.980 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 -- 172.21.15.59:0/1968231965 shutdown_connections 2023-12-19T18:39:28.980 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:28.408+0000 7f75f9b2f700 1 -- 172.21.15.59:0/1968231965 wait complete. 2023-12-19T18:39:28.980 INFO:teuthology.orchestra.run.smithi059.stdout:mgr epoch 5 is available 2023-12-19T18:39:28.980 INFO:teuthology.orchestra.run.smithi059.stdout:Setting orchestrator backend to cephadm... 2023-12-19T18:39:29.575 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:29 smithi059 ceph-mon[80661]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-19T18:39:29.575 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:29 smithi059 ceph-mon[80661]: mgrmap e7: smithi059.ccbnob(active, since 1.00554s) 2023-12-19T18:39:29.575 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:29 smithi059 ceph-mon[80661]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-19T18:39:30.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.352+0000 7fb3740bf700 1 Processor -- start 2023-12-19T18:39:30.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.352+0000 7fb3740bf700 1 -- start start 2023-12-19T18:39:30.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.352+0000 7fb3740bf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:30.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.352+0000 7fb3740bf700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c105420 con 0x7fb36c070fc0 2023-12-19T18:39:30.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.352+0000 7fb371e5b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:30.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.352+0000 7fb371e5b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38072/0 (socket says 172.21.15.59:38072) 2023-12-19T18:39:30.183 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.352+0000 7fb371e5b700 1 -- 172.21.15.59:0/2459684651 learned_addr learned my addr 172.21.15.59:0/2459684651 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.353+0000 7fb371e5b700 1 -- 172.21.15.59:0/2459684651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c105560 con 0x7fb36c070fc0 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.353+0000 7fb371e5b700 1 --2- 172.21.15.59:0/2459684651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c104ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3680098e0 tx=0x7fb368009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748724f656fc9287 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.353+0000 7fb370e59700 1 -- 172.21.15.59:0/2459684651 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb36800b860 con 0x7fb36c070fc0 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.353+0000 7fb370e59700 1 -- 172.21.15.59:0/2459684651 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb36800be30 con 0x7fb36c070fc0 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.353+0000 7fb370e59700 1 -- 172.21.15.59:0/2459684651 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb368010380 con 0x7fb36c070fc0 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.354+0000 7fb3740bf700 1 -- 172.21.15.59:0/2459684651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 msgr2=0x7fb36c104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.354+0000 7fb3740bf700 1 --2- 172.21.15.59:0/2459684651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c104ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3680098e0 tx=0x7fb368009250 comp rx=0 tx=0).stop 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.354+0000 7fb3740bf700 1 -- 172.21.15.59:0/2459684651 shutdown_connections 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.354+0000 7fb3740bf700 1 --2- 172.21.15.59:0/2459684651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c104ee0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:30.184 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.354+0000 7fb3740bf700 1 -- 172.21.15.59:0/2459684651 >> 172.21.15.59:0/2459684651 conn(0x7fb36c0fc970 msgr2=0x7fb36c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.354+0000 7fb3740bf700 1 -- 172.21.15.59:0/2459684651 shutdown_connections 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.354+0000 7fb3740bf700 1 -- 172.21.15.59:0/2459684651 wait complete. 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.354+0000 7fb3740bf700 1 Processor -- start 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb3740bf700 1 -- start start 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb3740bf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb3740bf700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c100de0 con 0x7fb36c070fc0 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb371e5b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb371e5b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c102960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38076/0 (socket says 172.21.15.59:38076) 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb371e5b700 1 -- 172.21.15.59:0/1419442973 learned_addr learned my addr 172.21.15.59:0/1419442973 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb371e5b700 1 -- 172.21.15.59:0/1419442973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3680094f0 con 0x7fb36c070fc0 2023-12-19T18:39:30.185 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb371e5b700 1 --2- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c102960 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb368006b00 tx=0x7fb368009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb3627fc700 1 -- 172.21.15.59:0/1419442973 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb368010a80 con 0x7fb36c070fc0 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb3627fc700 1 -- 172.21.15.59:0/1419442973 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb36801a430 con 0x7fb36c070fc0 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c100fe0 con 0x7fb36c070fc0 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.355+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c1013f0 con 0x7fb36c070fc0 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.356+0000 7fb3627fc700 1 -- 172.21.15.59:0/1419442973 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3680245b0 con 0x7fb36c070fc0 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.356+0000 7fb3627fc700 1 -- 172.21.15.59:0/1419442973 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7fb3680247e0 con 0x7fb36c070fc0 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.356+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36c04bb80 con 0x7fb36c070fc0 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.356+0000 7fb3627fc700 1 --2- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb358036510 0x7fb3580389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.357+0000 7fb3627fc700 1 -- 172.21.15.59:0/1419442973 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb36804aa50 con 0x7fb36c070fc0 2023-12-19T18:39:30.186 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.357+0000 7fb37165a700 1 --2- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb358036510 0x7fb3580389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:30.187 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.357+0000 7fb37165a700 1 --2- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb358036510 0x7fb3580389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009990 tx=0x7fb35c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:30.187 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.362+0000 7fb3627fc700 1 -- 172.21.15.59:0/1419442973 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb368018350 con 0x7fb36c070fc0 2023-12-19T18:39:30.187 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.575+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb36c10adb0 con 0x7fb358036510 2023-12-19T18:39:30.187 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.580+0000 7fb3627fc700 1 -- 172.21.15.59:0/1419442973 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb36c10adb0 con 0x7fb358036510 2023-12-19T18:39:30.187 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.582+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb358036510 msgr2=0x7fb3580389c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:30.187 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.582+0000 7fb3740bf700 1 --2- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb358036510 0x7fb3580389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009990 tx=0x7fb35c006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:30.188 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.583+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 msgr2=0x7fb36c102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:30.188 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.583+0000 7fb3740bf700 1 --2- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c102960 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb368006b00 tx=0x7fb368009250 comp rx=0 tx=0).stop 2023-12-19T18:39:30.188 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.583+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 shutdown_connections 2023-12-19T18:39:30.188 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.583+0000 7fb3740bf700 1 --2- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb358036510 0x7fb3580389c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:30.188 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.583+0000 7fb3740bf700 1 --2- 172.21.15.59:0/1419442973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb36c070fc0 0x7fb36c102960 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:30.188 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.583+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 >> 172.21.15.59:0/1419442973 conn(0x7fb36c0fc970 msgr2=0x7fb36c0fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:30.188 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.583+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 shutdown_connections 2023-12-19T18:39:30.188 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:29.583+0000 7fb3740bf700 1 -- 172.21.15.59:0/1419442973 wait complete. 2023-12-19T18:39:30.777 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:30 smithi059 ceph-mon[80661]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:30.777 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:30 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:30.777 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:30 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:39:30.777 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:30 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:30.777 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:30 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:39:30.777 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:30 smithi059 ceph-mon[80661]: mgrmap e8: smithi059.ccbnob(active, since 3s) 2023-12-19T18:39:31.320 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout value unchanged 2023-12-19T18:39:31.321 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.571+0000 7ff1b4670700 1 Processor -- start 2023-12-19T18:39:31.321 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.571+0000 7ff1b4670700 1 -- start start 2023-12-19T18:39:31.321 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.571+0000 7ff1b4670700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:31.321 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.571+0000 7ff1b4670700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac105440 con 0x7ff1ac070fa0 2023-12-19T18:39:31.321 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.572+0000 7ff1b240c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:31.321 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.572+0000 7ff1b240c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38084/0 (socket says 172.21.15.59:38084) 2023-12-19T18:39:31.321 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.572+0000 7ff1b240c700 1 -- 172.21.15.59:0/3887761711 learned_addr learned my addr 172.21.15.59:0/3887761711 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:31.321 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.572+0000 7ff1b240c700 1 -- 172.21.15.59:0/3887761711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac105580 con 0x7ff1ac070fa0 2023-12-19T18:39:31.321 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.572+0000 7ff1b240c700 1 --2- 172.21.15.59:0/3887761711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac104f00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009e50 tx=0x7ff19c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd23cfb63fa7e194 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.572+0000 7ff1b140a700 1 -- 172.21.15.59:0/3887761711 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff19c0092e0 con 0x7ff1ac070fa0 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.572+0000 7ff1b140a700 1 -- 172.21.15.59:0/3887761711 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff19c00be90 con 0x7ff1ac070fa0 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.572+0000 7ff1b140a700 1 -- 172.21.15.59:0/3887761711 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff19c007400 con 0x7ff1ac070fa0 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.573+0000 7ff1b4670700 1 -- 172.21.15.59:0/3887761711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 msgr2=0x7ff1ac104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.573+0000 7ff1b4670700 1 --2- 172.21.15.59:0/3887761711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac104f00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009e50 tx=0x7ff19c009840 comp rx=0 tx=0).stop 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.573+0000 7ff1b4670700 1 -- 172.21.15.59:0/3887761711 shutdown_connections 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.573+0000 7ff1b4670700 1 --2- 172.21.15.59:0/3887761711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac104f00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.573+0000 7ff1b4670700 1 -- 172.21.15.59:0/3887761711 >> 172.21.15.59:0/3887761711 conn(0x7ff1ac0fc970 msgr2=0x7ff1ac0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.573+0000 7ff1b4670700 1 -- 172.21.15.59:0/3887761711 shutdown_connections 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.573+0000 7ff1b4670700 1 -- 172.21.15.59:0/3887761711 wait complete. 2023-12-19T18:39:31.322 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1b4670700 1 Processor -- start 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1b4670700 1 -- start start 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1b4670700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac102980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1b4670700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac100e00 con 0x7ff1ac070fa0 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1b240c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1b240c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac102980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38092/0 (socket says 172.21.15.59:38092) 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1b240c700 1 -- 172.21.15.59:0/4130875547 learned_addr learned my addr 172.21.15.59:0/4130875547 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1b240c700 1 -- 172.21.15.59:0/4130875547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c009ae0 con 0x7ff1ac070fa0 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1b240c700 1 --2- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac102980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009fd0 tx=0x7ff19c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1aaffd700 1 -- 172.21.15.59:0/4130875547 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff19c007670 con 0x7ff1ac070fa0 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.574+0000 7ff1aaffd700 1 -- 172.21.15.59:0/4130875547 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff19c007c40 con 0x7ff1ac070fa0 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.575+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac101000 con 0x7ff1ac070fa0 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.575+0000 7ff1aaffd700 1 -- 172.21.15.59:0/4130875547 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff19c01cb60 con 0x7ff1ac070fa0 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.575+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac101470 con 0x7ff1ac070fa0 2023-12-19T18:39:31.323 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.575+0000 7ff1aaffd700 1 -- 172.21.15.59:0/4130875547 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7ff19c019070 con 0x7ff1ac070fa0 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.575+0000 7ff1aaffd700 1 --2- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff1940365f0 0x7ff194038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.576+0000 7ff1aaffd700 1 -- 172.21.15.59:0/4130875547 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff19c049f40 con 0x7ff1ac070fa0 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.576+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac10b140 con 0x7ff1ac070fa0 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.576+0000 7ff1b1c0b700 1 --2- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff1940365f0 0x7ff194038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.576+0000 7ff1b1c0b700 1 --2- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff1940365f0 0x7ff194038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009990 tx=0x7ff1a0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.581+0000 7ff1aaffd700 1 -- 172.21.15.59:0/4130875547 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff19c017080 con 0x7ff1ac070fa0 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.777+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1ac1016b0 con 0x7ff1940365f0 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.778+0000 7ff1aaffd700 1 -- 172.21.15.59:0/4130875547 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7ff1ac1016b0 con 0x7ff1940365f0 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff1940365f0 msgr2=0x7ff194038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 --2- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff1940365f0 0x7ff194038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009990 tx=0x7ff1a0006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 msgr2=0x7ff1ac102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 --2- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac102980 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009fd0 tx=0x7ff19c009840 comp rx=0 tx=0).stop 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 shutdown_connections 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 --2- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff1940365f0 0x7ff194038aa0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:31.324 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 --2- 172.21.15.59:0/4130875547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff1ac070fa0 0x7ff1ac102980 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:31.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 >> 172.21.15.59:0/4130875547 conn(0x7ff1ac0fc970 msgr2=0x7ff1ac0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:31.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 shutdown_connections 2023-12-19T18:39:31.325 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:30.780+0000 7ff1b4670700 1 -- 172.21.15.59:0/4130875547 wait complete. 2023-12-19T18:39:31.325 INFO:teuthology.orchestra.run.smithi059.stdout:Generating ssh key... 2023-12-19T18:39:31.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:31 smithi059 ceph-mon[80661]: [19/Dec/2023:18:39:29] ENGINE Bus STARTING 2023-12-19T18:39:31.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:31 smithi059 ceph-mon[80661]: [19/Dec/2023:18:39:29] ENGINE Serving on https://172.21.15.59:7150 2023-12-19T18:39:31.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:31 smithi059 ceph-mon[80661]: [19/Dec/2023:18:39:29] ENGINE Bus STARTED 2023-12-19T18:39:32.844 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:32 smithi059 ceph-mon[80661]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:32.844 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:32 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:32.844 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:32 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:32.852 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.806+0000 7f19ee8de700 1 Processor -- start 2023-12-19T18:39:32.852 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.806+0000 7f19ee8de700 1 -- start start 2023-12-19T18:39:32.853 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.806+0000 7f19ee8de700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e8103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:32.853 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.806+0000 7f19ee8de700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e8104310 con 0x7f19e81039c0 2023-12-19T18:39:32.853 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.806+0000 7f19e7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e8103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:32.853 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.806+0000 7f19e7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e8103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41308/0 (socket says 172.21.15.59:41308) 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.806+0000 7f19e7fff700 1 -- 172.21.15.59:0/4000581513 learned_addr learned my addr 172.21.15.59:0/4000581513 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.806+0000 7f19e7fff700 1 -- 172.21.15.59:0/4000581513 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e8101fb0 con 0x7f19e81039c0 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.806+0000 7f19e7fff700 1 --2- 172.21.15.59:0/4000581513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e8103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f19d80098e0 tx=0x7f19d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4043cac7445dbb0a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19e6ffd700 1 -- 172.21.15.59:0/4000581513 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d800b860 con 0x7f19e81039c0 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19e6ffd700 1 -- 172.21.15.59:0/4000581513 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f19d800be30 con 0x7f19e81039c0 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19e6ffd700 1 -- 172.21.15.59:0/4000581513 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d8010380 con 0x7f19e81039c0 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19ee8de700 1 -- 172.21.15.59:0/4000581513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 msgr2=0x7f19e8103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19ee8de700 1 --2- 172.21.15.59:0/4000581513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e8103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f19d80098e0 tx=0x7f19d8009250 comp rx=0 tx=0).stop 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19ee8de700 1 -- 172.21.15.59:0/4000581513 shutdown_connections 2023-12-19T18:39:32.854 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19ee8de700 1 --2- 172.21.15.59:0/4000581513 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e8103dd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19ee8de700 1 -- 172.21.15.59:0/4000581513 >> 172.21.15.59:0/4000581513 conn(0x7f19e80fda40 msgr2=0x7f19e80ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19ee8de700 1 -- 172.21.15.59:0/4000581513 shutdown_connections 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.807+0000 7f19ee8de700 1 -- 172.21.15.59:0/4000581513 wait complete. 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.808+0000 7f19ee8de700 1 Processor -- start 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.808+0000 7f19ee8de700 1 -- start start 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.808+0000 7f19ee8de700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e81110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.808+0000 7f19ee8de700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e8111630 con 0x7f19e81039c0 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.808+0000 7f19e7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e81110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.808+0000 7f19e7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e81110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41314/0 (socket says 172.21.15.59:41314) 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.808+0000 7f19e7fff700 1 -- 172.21.15.59:0/3883984577 learned_addr learned my addr 172.21.15.59:0/3883984577 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.809+0000 7f19e7fff700 1 -- 172.21.15.59:0/3883984577 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d80094f0 con 0x7f19e81039c0 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.809+0000 7f19e7fff700 1 --2- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e81110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f19d8006b00 tx=0x7f19d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.809+0000 7f19e4ff9700 1 -- 172.21.15.59:0/3883984577 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d80109b0 con 0x7f19e81039c0 2023-12-19T18:39:32.855 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.809+0000 7f19e4ff9700 1 -- 172.21.15.59:0/3883984577 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f19d801a430 con 0x7f19e81039c0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.809+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e8111830 con 0x7f19e81039c0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.809+0000 7f19e4ff9700 1 -- 172.21.15.59:0/3883984577 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19d80245b0 con 0x7f19e81039c0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.809+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e8111c40 con 0x7f19e81039c0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.810+0000 7f19e4ff9700 1 -- 172.21.15.59:0/3883984577 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f19d80247e0 con 0x7f19e81039c0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.810+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e8102d60 con 0x7f19e81039c0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.810+0000 7f19e4ff9700 1 --2- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f19d4036640 0x7f19d4038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.810+0000 7f19e4ff9700 1 -- 172.21.15.59:0/3883984577 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f19d804ab80 con 0x7f19e81039c0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.810+0000 7f19e77fe700 1 --2- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f19d4036640 0x7f19d4038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.811+0000 7f19e77fe700 1 --2- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f19d4036640 0x7f19d4038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009990 tx=0x7f19d0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:31.815+0000 7f19e4ff9700 1 -- 172.21.15.59:0/3883984577 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f19d8018350 con 0x7f19e81039c0 2023-12-19T18:39:32.856 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.014+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f19e810adf0 con 0x7f19d4036640 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.303+0000 7f19e4ff9700 1 -- 172.21.15.59:0/3883984577 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f19e810adf0 con 0x7f19d4036640 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f19d4036640 msgr2=0x7f19d4038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 --2- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f19d4036640 0x7f19d4038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009990 tx=0x7f19d0006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 msgr2=0x7f19e81110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 --2- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e81110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f19d8006b00 tx=0x7f19d8009250 comp rx=0 tx=0).stop 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 shutdown_connections 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 --2- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f19d4036640 0x7f19d4038af0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 --2- 172.21.15.59:0/3883984577 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19e81039c0 0x7f19e81110f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 >> 172.21.15.59:0/3883984577 conn(0x7f19e80fda40 msgr2=0x7f19e80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 shutdown_connections 2023-12-19T18:39:32.857 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:32.305+0000 7f19ee8de700 1 -- 172.21.15.59:0/3883984577 wait complete. 2023-12-19T18:39:33.941 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:33 smithi059 ceph-mon[80661]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:33.941 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:33 smithi059 ceph-mon[80661]: Generating ssh key... 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcZ741Eu8uRFJIzSVCG5g7wC+O6CkKG23f2dBASTDklVCRhViodR92fqpcBApSuyqD81p6nRoUvVgu29pS5q4J2KCWq2plNgEBnbBwinjbdjdmpFEPde+ze6U4vl2KcTs7kkK6SqHoZHwd8JkNzM4x+Fr+ijaNFVeyQzoNALsX2K5IYvpS7/PHkmN7nmhjNWpzsbyqDa0hWiC8qc0fWon4j1eS0pNV3RRuSrUdMAAAgPZMEZelXAmZRkgaXKjzCglomn2Ig1BcRiydUM+1iUz9Mhu6EAlvtFBDKT4VtgmQCs4Jn2tfOl2rti482QZaCp9iNhsO1q6G36O+nTN6vebHW5l6IcmegfozshZa8/s4Zx61r+XWmKGFOZjdOHxt4qp3VXMnlN/8aPTmd1oPpQHFID4p7zGlysKcj85L3iGoL0WIRvf+g1uOVSddpUJvAx+zvd0dSyRKZLyw+4ZiJvHImwyVeh8DSELGOeN/uWzFilrQ4lHBl1Lm/IYGtHPswhM= ceph-bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.263+0000 7ff4d6ff5700 1 Processor -- start 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.263+0000 7ff4d6ff5700 1 -- start start 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.263+0000 7ff4d6ff5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d01015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.263+0000 7ff4d6ff5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d0101b20 con 0x7ff4d01011d0 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.263+0000 7ff4d4d91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d01015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.263+0000 7ff4d4d91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d01015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41320/0 (socket says 172.21.15.59:41320) 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.263+0000 7ff4d4d91700 1 -- 172.21.15.59:0/114999926 learned_addr learned my addr 172.21.15.59:0/114999926 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.263+0000 7ff4d4d91700 1 -- 172.21.15.59:0/114999926 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d0101c60 con 0x7ff4d01011d0 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4d4d91700 1 --2- 172.21.15.59:0/114999926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d01015e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40098e0 tx=0x7ff4c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6881c8fa9f5dfb5a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:33.949 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4cf7fe700 1 -- 172.21.15.59:0/114999926 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c400b860 con 0x7ff4d01011d0 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4cf7fe700 1 -- 172.21.15.59:0/114999926 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff4c400be30 con 0x7ff4d01011d0 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4cf7fe700 1 -- 172.21.15.59:0/114999926 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c4010380 con 0x7ff4d01011d0 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/114999926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 msgr2=0x7ff4d01015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4d6ff5700 1 --2- 172.21.15.59:0/114999926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d01015e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40098e0 tx=0x7ff4c4009250 comp rx=0 tx=0).stop 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/114999926 shutdown_connections 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4d6ff5700 1 --2- 172.21.15.59:0/114999926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d01015e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/114999926 >> 172.21.15.59:0/114999926 conn(0x7ff4d00fc950 msgr2=0x7ff4d00fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/114999926 shutdown_connections 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.264+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/114999926 wait complete. 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.265+0000 7ff4d6ff5700 1 Processor -- start 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.265+0000 7ff4d6ff5700 1 -- start start 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.265+0000 7ff4d6ff5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d010ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.265+0000 7ff4d6ff5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d010f480 con 0x7ff4d01011d0 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.265+0000 7ff4d4d91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d010ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:33.950 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.265+0000 7ff4d4d91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d010ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41322/0 (socket says 172.21.15.59:41322) 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.266+0000 7ff4d4d91700 1 -- 172.21.15.59:0/3396420963 learned_addr learned my addr 172.21.15.59:0/3396420963 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.266+0000 7ff4d4d91700 1 -- 172.21.15.59:0/3396420963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c40094f0 con 0x7ff4d01011d0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.266+0000 7ff4d4d91700 1 --2- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d010ef40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4006b00 tx=0x7ff4c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.266+0000 7ff4cd7fa700 1 -- 172.21.15.59:0/3396420963 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c40109b0 con 0x7ff4d01011d0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.266+0000 7ff4cd7fa700 1 -- 172.21.15.59:0/3396420963 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff4c4024440 con 0x7ff4d01011d0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.266+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d010f680 con 0x7ff4d01011d0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.266+0000 7ff4cd7fa700 1 -- 172.21.15.59:0/3396420963 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c401a5f0 con 0x7ff4d01011d0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.266+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d010fa90 con 0x7ff4d01011d0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.267+0000 7ff4cd7fa700 1 -- 172.21.15.59:0/3396420963 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7ff4c401a870 con 0x7ff4d01011d0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.267+0000 7ff4cd7fa700 1 --2- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff4c0036630 0x7ff4c0038ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.267+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d004bb80 con 0x7ff4d01011d0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.267+0000 7ff4cd7fa700 1 -- 172.21.15.59:0/3396420963 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff4c404a5e0 con 0x7ff4d01011d0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.267+0000 7ff4cffff700 1 --2- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff4c0036630 0x7ff4c0038ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.268+0000 7ff4cffff700 1 --2- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff4c0036630 0x7ff4c0038ae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009990 tx=0x7ff4bc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:33.951 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.272+0000 7ff4cd7fa700 1 -- 172.21.15.59:0/3396420963 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff4c4022340 con 0x7ff4d01011d0 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.467+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4d0108c30 con 0x7ff4c0036630 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.468+0000 7ff4cd7fa700 1 -- 172.21.15.59:0/3396420963 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7ff4d0108c30 con 0x7ff4c0036630 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff4c0036630 msgr2=0x7ff4c0038ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 --2- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff4c0036630 0x7ff4c0038ae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009990 tx=0x7ff4bc006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 msgr2=0x7ff4d010ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 --2- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d010ef40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4006b00 tx=0x7ff4c4009250 comp rx=0 tx=0).stop 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 shutdown_connections 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 --2- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff4c0036630 0x7ff4c0038ae0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 --2- 172.21.15.59:0/3396420963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff4d01011d0 0x7ff4d010ef40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 >> 172.21.15.59:0/3396420963 conn(0x7ff4d00fc950 msgr2=0x7ff4d0105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 shutdown_connections 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:33.470+0000 7ff4d6ff5700 1 -- 172.21.15.59:0/3396420963 wait complete. 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:Adding key to root@localhost authorized_keys... 2023-12-19T18:39:33.952 INFO:teuthology.orchestra.run.smithi059.stdout:Adding host smithi059... 2023-12-19T18:39:34.864 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:34 smithi059 ceph-mon[80661]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:35.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:35 smithi059 ceph-mon[80661]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:36.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:36 smithi059 ceph-mon[80661]: Deploying cephadm binary to smithi059 2023-12-19T18:39:37.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Added host 'smithi059' with addr '172.21.15.59' 2023-12-19T18:39:37.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.307+0000 7fb3a09f1700 1 Processor -- start 2023-12-19T18:39:37.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.307+0000 7fb3a09f1700 1 -- start start 2023-12-19T18:39:37.207 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.307+0000 7fb3a09f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c103da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.307+0000 7fb3a09f1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c1042e0 con 0x7fb39c103990 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.307+0000 7fb39a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c103da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.307+0000 7fb39a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c103da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41336/0 (socket says 172.21.15.59:41336) 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.307+0000 7fb39a59c700 1 -- 172.21.15.59:0/3648455629 learned_addr learned my addr 172.21.15.59:0/3648455629 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.308+0000 7fb39a59c700 1 -- 172.21.15.59:0/3648455629 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c104420 con 0x7fb39c103990 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.308+0000 7fb39a59c700 1 --2- 172.21.15.59:0/3648455629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c103da0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0098e0 tx=0x7fb38c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f10710964c4830 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.308+0000 7fb39959a700 1 -- 172.21.15.59:0/3648455629 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb38c00b860 con 0x7fb39c103990 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.308+0000 7fb39959a700 1 -- 172.21.15.59:0/3648455629 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb38c00be30 con 0x7fb39c103990 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.308+0000 7fb39959a700 1 -- 172.21.15.59:0/3648455629 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb38c010380 con 0x7fb39c103990 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.309+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3648455629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 msgr2=0x7fb39c103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.309+0000 7fb3a09f1700 1 --2- 172.21.15.59:0/3648455629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c103da0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0098e0 tx=0x7fb38c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.309+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3648455629 shutdown_connections 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.309+0000 7fb3a09f1700 1 --2- 172.21.15.59:0/3648455629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c103da0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.309+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3648455629 >> 172.21.15.59:0/3648455629 conn(0x7fb39c0fc0b0 msgr2=0x7fb39c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:37.208 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.309+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3648455629 shutdown_connections 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.309+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3648455629 wait complete. 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.309+0000 7fb3a09f1700 1 Processor -- start 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.309+0000 7fb3a09f1700 1 -- start start 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb3a09f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c111110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb3a09f1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c111650 con 0x7fb39c103990 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb39a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c111110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb39a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c111110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41352/0 (socket says 172.21.15.59:41352) 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb39a59c700 1 -- 172.21.15.59:0/3336367046 learned_addr learned my addr 172.21.15.59:0/3336367046 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb39a59c700 1 -- 172.21.15.59:0/3336367046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0094f0 con 0x7fb39c103990 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb39a59c700 1 --2- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c111110 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb38c006b00 tx=0x7fb38c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb392ffd700 1 -- 172.21.15.59:0/3336367046 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb38c0109b0 con 0x7fb39c103990 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb392ffd700 1 -- 172.21.15.59:0/3336367046 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb38c024440 con 0x7fb39c103990 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c111850 con 0x7fb39c103990 2023-12-19T18:39:37.209 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb392ffd700 1 -- 172.21.15.59:0/3336367046 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb38c01a5f0 con 0x7fb39c103990 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.310+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c111c60 con 0x7fb39c103990 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.311+0000 7fb392ffd700 1 -- 172.21.15.59:0/3336367046 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fb38c024990 con 0x7fb39c103990 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.311+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c04bb80 con 0x7fb39c103990 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.311+0000 7fb392ffd700 1 --2- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb388036640 0x7fb388038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.311+0000 7fb392ffd700 1 -- 172.21.15.59:0/3336367046 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb38c04a570 con 0x7fb39c103990 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.312+0000 7fb399d9b700 1 --2- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb388036640 0x7fb388038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.312+0000 7fb399d9b700 1 --2- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb388036640 0x7fb388038af0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb384009990 tx=0x7fb384006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.316+0000 7fb392ffd700 1 -- 172.21.15.59:0/3336367046 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb38c006d70 con 0x7fb39c103990 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:34.518+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}) v1 -- 0x7fb39c10afe0 con 0x7fb388036640 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.600+0000 7fb392ffd700 1 -- 172.21.15.59:0/3336367046 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fb39c10afe0 con 0x7fb388036640 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.602+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb388036640 msgr2=0x7fb388038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.602+0000 7fb3a09f1700 1 --2- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb388036640 0x7fb388038af0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb384009990 tx=0x7fb384006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.603+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 msgr2=0x7fb39c111110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.603+0000 7fb3a09f1700 1 --2- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c111110 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb38c006b00 tx=0x7fb38c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:37.210 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.603+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 shutdown_connections 2023-12-19T18:39:37.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.603+0000 7fb3a09f1700 1 --2- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb388036640 0x7fb388038af0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:37.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.603+0000 7fb3a09f1700 1 --2- 172.21.15.59:0/3336367046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb39c103990 0x7fb39c111110 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:37.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.603+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 >> 172.21.15.59:0/3336367046 conn(0x7fb39c0fc0b0 msgr2=0x7fb39c0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:37.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.603+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 shutdown_connections 2023-12-19T18:39:37.211 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:36.603+0000 7fb3a09f1700 1 -- 172.21.15.59:0/3336367046 wait complete. 2023-12-19T18:39:37.211 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying mon service with default placement... 2023-12-19T18:39:37.862 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:37 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:37.862 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:37 smithi059 ceph-mon[80661]: Added host smithi059 2023-12-19T18:39:37.863 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:37 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:39:38.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-12-19T18:39:38.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.639+0000 7fc89aa9a700 1 Processor -- start 2023-12-19T18:39:38.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.639+0000 7fc89aa9a700 1 -- start start 2023-12-19T18:39:38.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.639+0000 7fc89aa9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc894104f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:38.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.639+0000 7fc89aa9a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc894105480 con 0x7fc894076bc0 2023-12-19T18:39:38.601 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.639+0000 7fc898836700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc894104f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.640+0000 7fc898836700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc894104f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41360/0 (socket says 172.21.15.59:41360) 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.640+0000 7fc898836700 1 -- 172.21.15.59:0/124738728 learned_addr learned my addr 172.21.15.59:0/124738728 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.640+0000 7fc898836700 1 -- 172.21.15.59:0/124738728 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8941055c0 con 0x7fc894076bc0 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.640+0000 7fc898836700 1 --2- 172.21.15.59:0/124738728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc894104f40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00fa00 tx=0x7fc87c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7c2d7d332dc24 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.640+0000 7fc8937fe700 1 -- 172.21.15.59:0/124738728 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87c009cb0 con 0x7fc894076bc0 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.640+0000 7fc8937fe700 1 -- 172.21.15.59:0/124738728 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fc87c009e10 con 0x7fc894076bc0 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.640+0000 7fc8937fe700 1 -- 172.21.15.59:0/124738728 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87c016730 con 0x7fc894076bc0 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.641+0000 7fc89aa9a700 1 -- 172.21.15.59:0/124738728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 msgr2=0x7fc894104f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.641+0000 7fc89aa9a700 1 --2- 172.21.15.59:0/124738728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc894104f40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00fa00 tx=0x7fc87c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.641+0000 7fc89aa9a700 1 -- 172.21.15.59:0/124738728 shutdown_connections 2023-12-19T18:39:38.602 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.641+0000 7fc89aa9a700 1 --2- 172.21.15.59:0/124738728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc894104f40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.641+0000 7fc89aa9a700 1 -- 172.21.15.59:0/124738728 >> 172.21.15.59:0/124738728 conn(0x7fc8940fc950 msgr2=0x7fc8940fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.641+0000 7fc89aa9a700 1 -- 172.21.15.59:0/124738728 shutdown_connections 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.641+0000 7fc89aa9a700 1 -- 172.21.15.59:0/124738728 wait complete. 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.641+0000 7fc89aa9a700 1 Processor -- start 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.641+0000 7fc89aa9a700 1 -- start start 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc89aa9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc89410eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc89aa9a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89410f420 con 0x7fc894076bc0 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc898836700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc89410eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc898836700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc89410eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41364/0 (socket says 172.21.15.59:41364) 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc898836700 1 -- 172.21.15.59:0/3207959711 learned_addr learned my addr 172.21.15.59:0/3207959711 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc898836700 1 -- 172.21.15.59:0/3207959711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c00f610 con 0x7fc894076bc0 2023-12-19T18:39:38.603 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc898836700 1 --2- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc89410eee0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc87c006fd0 tx=0x7fc87c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc8917fa700 1 -- 172.21.15.59:0/3207959711 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87c0177c0 con 0x7fc894076bc0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc8917fa700 1 -- 172.21.15.59:0/3207959711 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fc87c017920 con 0x7fc894076bc0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89410f620 con 0x7fc894076bc0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc8917fa700 1 -- 172.21.15.59:0/3207959711 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87c01f590 con 0x7fc894076bc0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.642+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89410fa30 con 0x7fc894076bc0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.643+0000 7fc8917fa700 1 -- 172.21.15.59:0/3207959711 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fc87c020470 con 0x7fc894076bc0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.643+0000 7fc8917fa700 1 --2- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fc8800365f0 0x7fc880038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.643+0000 7fc8917fa700 1 -- 172.21.15.59:0/3207959711 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc87c04df30 con 0x7fc894076bc0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.643+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89404bb80 con 0x7fc894076bc0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.643+0000 7fc893fff700 1 --2- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fc8800365f0 0x7fc880038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:38.604 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.644+0000 7fc893fff700 1 --2- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fc8800365f0 0x7fc880038aa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc884009990 tx=0x7fc884006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.649+0000 7fc8917fa700 1 -- 172.21.15.59:0/3207959711 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc87c024080 con 0x7fc894076bc0 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.862+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fc894108d30 con 0x7fc8800365f0 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.869+0000 7fc8917fa700 1 -- 172.21.15.59:0/3207959711 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc894108d30 con 0x7fc8800365f0 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fc8800365f0 msgr2=0x7fc880038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 --2- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fc8800365f0 0x7fc880038aa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc884009990 tx=0x7fc884006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 msgr2=0x7fc89410eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 --2- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc89410eee0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc87c006fd0 tx=0x7fc87c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 shutdown_connections 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 --2- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fc8800365f0 0x7fc880038aa0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 --2- 172.21.15.59:0/3207959711 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc894076bc0 0x7fc89410eee0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:38.605 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 >> 172.21.15.59:0/3207959711 conn(0x7fc8940fc950 msgr2=0x7fc894103100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:38.606 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 shutdown_connections 2023-12-19T18:39:38.606 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:37.872+0000 7fc89aa9a700 1 -- 172.21.15.59:0/3207959711 wait complete. 2023-12-19T18:39:38.606 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying mgr service with default placement... 2023-12-19T18:39:39.149 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:38 smithi059 ceph-mon[80661]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:39.149 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:38 smithi059 ceph-mon[80661]: Saving service mon spec with placement count:5 2023-12-19T18:39:39.149 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:38 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:40.870 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:40 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:40.871 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:40 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:40.871 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:40 smithi059 ceph-mon[80661]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:40.871 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:40 smithi059 ceph-mon[80661]: Saving service mgr spec with placement count:2 2023-12-19T18:39:40.871 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:40 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:40.878 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-12-19T18:39:40.878 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.084+0000 7fa928e3e700 1 Processor -- start 2023-12-19T18:39:40.878 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.084+0000 7fa928e3e700 1 -- start start 2023-12-19T18:39:40.878 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.085+0000 7fa928e3e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa9241026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.085+0000 7fa928e3e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa924102c00 con 0x7fa9241022b0 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.085+0000 7fa92259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa9241026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.085+0000 7fa92259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa9241026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41378/0 (socket says 172.21.15.59:41378) 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.085+0000 7fa92259c700 1 -- 172.21.15.59:0/3773565665 learned_addr learned my addr 172.21.15.59:0/3773565665 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.085+0000 7fa92259c700 1 -- 172.21.15.59:0/3773565665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa924102d40 con 0x7fa9241022b0 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.085+0000 7fa92259c700 1 --2- 172.21.15.59:0/3773565665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa9241026c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa9140098e0 tx=0x7fa914009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e92dad37c6ddca6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.085+0000 7fa92159a700 1 -- 172.21.15.59:0/3773565665 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa91400b860 con 0x7fa9241022b0 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.086+0000 7fa92159a700 1 -- 172.21.15.59:0/3773565665 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa91400be70 con 0x7fa9241022b0 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.086+0000 7fa92159a700 1 -- 172.21.15.59:0/3773565665 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa914010380 con 0x7fa9241022b0 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.086+0000 7fa928e3e700 1 -- 172.21.15.59:0/3773565665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 msgr2=0x7fa9241026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.086+0000 7fa928e3e700 1 --2- 172.21.15.59:0/3773565665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa9241026c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa9140098e0 tx=0x7fa914009250 comp rx=0 tx=0).stop 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.087+0000 7fa928e3e700 1 -- 172.21.15.59:0/3773565665 shutdown_connections 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.087+0000 7fa928e3e700 1 --2- 172.21.15.59:0/3773565665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa9241026c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:40.879 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.087+0000 7fa928e3e700 1 -- 172.21.15.59:0/3773565665 >> 172.21.15.59:0/3773565665 conn(0x7fa9240fda10 msgr2=0x7fa9240ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.087+0000 7fa928e3e700 1 -- 172.21.15.59:0/3773565665 shutdown_connections 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.087+0000 7fa928e3e700 1 -- 172.21.15.59:0/3773565665 wait complete. 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.087+0000 7fa928e3e700 1 Processor -- start 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.087+0000 7fa928e3e700 1 -- start start 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.087+0000 7fa928e3e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa924078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.087+0000 7fa928e3e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa924077190 con 0x7fa9241022b0 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa92259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa924078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa92259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa924078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41382/0 (socket says 172.21.15.59:41382) 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa92259c700 1 -- 172.21.15.59:0/2853905429 learned_addr learned my addr 172.21.15.59:0/2853905429 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa92259c700 1 -- 172.21.15.59:0/2853905429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9140094f0 con 0x7fa9241022b0 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa92259c700 1 --2- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa924078d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa914011b60 tx=0x7fa914009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa912ffd700 1 -- 172.21.15.59:0/2853905429 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa914010850 con 0x7fa9241022b0 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa912ffd700 1 -- 172.21.15.59:0/2853905429 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa914010e60 con 0x7fa9241022b0 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa924077390 con 0x7fa9241022b0 2023-12-19T18:39:40.880 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa912ffd700 1 -- 172.21.15.59:0/2853905429 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa91401ae20 con 0x7fa9241022b0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.088+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9240777a0 con 0x7fa9241022b0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.089+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92404bb80 con 0x7fa9241022b0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.089+0000 7fa912ffd700 1 -- 172.21.15.59:0/2853905429 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fa9140109b0 con 0x7fa9241022b0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.090+0000 7fa912ffd700 1 --2- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa90c0365f0 0x7fa90c038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.090+0000 7fa912ffd700 1 -- 172.21.15.59:0/2853905429 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa91404a210 con 0x7fa9241022b0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.090+0000 7fa921d9b700 1 --2- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa90c0365f0 0x7fa90c038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.090+0000 7fa921d9b700 1 --2- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa90c0365f0 0x7fa90c038aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa918009990 tx=0x7fa918006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.097+0000 7fa912ffd700 1 -- 172.21.15.59:0/2853905429 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa91401a6a0 con 0x7fa9241022b0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.305+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fa92410ade0 con 0x7fa90c0365f0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.310+0000 7fa912ffd700 1 -- 172.21.15.59:0/2853905429 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa92410ade0 con 0x7fa90c0365f0 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.312+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa90c0365f0 msgr2=0x7fa90c038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.312+0000 7fa928e3e700 1 --2- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa90c0365f0 0x7fa90c038aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa918009990 tx=0x7fa918006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.312+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 msgr2=0x7fa924078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:40.881 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.312+0000 7fa928e3e700 1 --2- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa924078d10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa914011b60 tx=0x7fa914009250 comp rx=0 tx=0).stop 2023-12-19T18:39:40.882 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.312+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 shutdown_connections 2023-12-19T18:39:40.882 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.312+0000 7fa928e3e700 1 --2- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa90c0365f0 0x7fa90c038aa0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:40.882 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.312+0000 7fa928e3e700 1 --2- 172.21.15.59:0/2853905429 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9241022b0 0x7fa924078d10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:40.882 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.312+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 >> 172.21.15.59:0/2853905429 conn(0x7fa9240fda10 msgr2=0x7fa9240ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:40.882 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.313+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 shutdown_connections 2023-12-19T18:39:40.882 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:40.313+0000 7fa928e3e700 1 -- 172.21.15.59:0/2853905429 wait complete. 2023-12-19T18:39:40.882 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying crash service with default placement... 2023-12-19T18:39:42.581 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2023-12-19T18:39:42.581 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.284+0000 7f3c91542700 1 Processor -- start 2023-12-19T18:39:42.581 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.284+0000 7f3c91542700 1 -- start start 2023-12-19T18:39:42.581 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.284+0000 7f3c91542700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c1015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:42.581 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.284+0000 7f3c91542700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c101b20 con 0x7f3c8c1011d0 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.284+0000 7f3c8affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c1015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.284+0000 7f3c8affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c1015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41396/0 (socket says 172.21.15.59:41396) 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.284+0000 7f3c8affd700 1 -- 172.21.15.59:0/1541469192 learned_addr learned my addr 172.21.15.59:0/1541469192 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.285+0000 7f3c8affd700 1 -- 172.21.15.59:0/1541469192 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c101c60 con 0x7f3c8c1011d0 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.285+0000 7f3c8affd700 1 --2- 172.21.15.59:0/1541469192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c1015e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c0098e0 tx=0x7f3c7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc8d7fbc29784c5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.285+0000 7f3c89ffb700 1 -- 172.21.15.59:0/1541469192 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7c00b860 con 0x7f3c8c1011d0 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.285+0000 7f3c89ffb700 1 -- 172.21.15.59:0/1541469192 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c7c00be70 con 0x7f3c8c1011d0 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.285+0000 7f3c89ffb700 1 -- 172.21.15.59:0/1541469192 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7c010380 con 0x7f3c8c1011d0 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.286+0000 7f3c91542700 1 -- 172.21.15.59:0/1541469192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 msgr2=0x7f3c8c1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.286+0000 7f3c91542700 1 --2- 172.21.15.59:0/1541469192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c1015e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c0098e0 tx=0x7f3c7c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.286+0000 7f3c91542700 1 -- 172.21.15.59:0/1541469192 shutdown_connections 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.286+0000 7f3c91542700 1 --2- 172.21.15.59:0/1541469192 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c1015e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.286+0000 7f3c91542700 1 -- 172.21.15.59:0/1541469192 >> 172.21.15.59:0/1541469192 conn(0x7f3c8c0fc970 msgr2=0x7f3c8c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:42.582 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.286+0000 7f3c91542700 1 -- 172.21.15.59:0/1541469192 shutdown_connections 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.286+0000 7f3c91542700 1 -- 172.21.15.59:0/1541469192 wait complete. 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.287+0000 7f3c91542700 1 Processor -- start 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.287+0000 7f3c91542700 1 -- start start 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.287+0000 7f3c91542700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c111190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.287+0000 7f3c91542700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c1116d0 con 0x7f3c8c1011d0 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.287+0000 7f3c8affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c111190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.287+0000 7f3c8affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c111190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41400/0 (socket says 172.21.15.59:41400) 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.287+0000 7f3c8affd700 1 -- 172.21.15.59:0/4120643838 learned_addr learned my addr 172.21.15.59:0/4120643838 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.287+0000 7f3c8affd700 1 -- 172.21.15.59:0/4120643838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c0094f0 con 0x7f3c8c1011d0 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.287+0000 7f3c8affd700 1 --2- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c111190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c011b60 tx=0x7f3c7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.288+0000 7f3c737fe700 1 -- 172.21.15.59:0/4120643838 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7c010850 con 0x7f3c8c1011d0 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.288+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8c1118d0 con 0x7f3c8c1011d0 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.288+0000 7f3c737fe700 1 -- 172.21.15.59:0/4120643838 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c7c010e60 con 0x7f3c8c1011d0 2023-12-19T18:39:42.583 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.288+0000 7f3c737fe700 1 -- 172.21.15.59:0/4120643838 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c7c01ae20 con 0x7f3c8c1011d0 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.288+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8c111ce0 con 0x7f3c8c1011d0 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.289+0000 7f3c737fe700 1 -- 172.21.15.59:0/4120643838 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f3c7c02d3f0 con 0x7f3c8c1011d0 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.289+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c04bb80 con 0x7f3c8c1011d0 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.289+0000 7f3c737fe700 1 --2- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f3c74036640 0x7f3c74038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.289+0000 7f3c737fe700 1 -- 172.21.15.59:0/4120643838 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3c7c04a1b0 con 0x7f3c8c1011d0 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.289+0000 7f3c8a7fc700 1 --2- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f3c74036640 0x7f3c74038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.290+0000 7f3c8a7fc700 1 --2- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f3c74036640 0x7f3c74038af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009990 tx=0x7f3c80006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.294+0000 7f3c737fe700 1 -- 172.21.15.59:0/4120643838 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3c7c02b8e0 con 0x7f3c8c1011d0 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.511+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c8c0febe0 con 0x7f3c74036640 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.515+0000 7f3c737fe700 1 -- 172.21.15.59:0/4120643838 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f3c8c0febe0 con 0x7f3c74036640 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f3c74036640 msgr2=0x7f3c74038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 --2- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f3c74036640 0x7f3c74038af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009990 tx=0x7f3c80006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 msgr2=0x7f3c8c111190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:42.584 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 --2- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c111190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c011b60 tx=0x7f3c7c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:42.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 shutdown_connections 2023-12-19T18:39:42.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 --2- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f3c74036640 0x7f3c74038af0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:42.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 --2- 172.21.15.59:0/4120643838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3c8c1011d0 0x7f3c8c111190 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:42.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 >> 172.21.15.59:0/4120643838 conn(0x7f3c8c0fc970 msgr2=0x7f3c8c070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:42.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 shutdown_connections 2023-12-19T18:39:42.585 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:41.518+0000 7f3c91542700 1 -- 172.21.15.59:0/4120643838 wait complete. 2023-12-19T18:39:42.585 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying prometheus service with default placement... 2023-12-19T18:39:42.816 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:42 smithi059 ceph-mon[80661]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:42.816 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:42 smithi059 ceph-mon[80661]: Saving service crash spec with placement * 2023-12-19T18:39:42.816 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:42 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:42.816 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:42 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:43.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:43 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:43.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:43 smithi059 ceph-mon[80661]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:43.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:43 smithi059 ceph-mon[80661]: Saving service prometheus spec with placement count:1 2023-12-19T18:39:43.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:43 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2023-12-19T18:39:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.038+0000 7fa890357700 1 Processor -- start 2023-12-19T18:39:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.038+0000 7fa890357700 1 -- start start 2023-12-19T18:39:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.038+0000 7fa890357700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.038+0000 7fa890357700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa888104310 con 0x7fa8881039c0 2023-12-19T18:39:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.038+0000 7fa88e0f3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.038+0000 7fa88e0f3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45550/0 (socket says 172.21.15.59:45550) 2023-12-19T18:39:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.038+0000 7fa88e0f3700 1 -- 172.21.15.59:0/4215767015 learned_addr learned my addr 172.21.15.59:0/4215767015 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:44.576 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.039+0000 7fa88e0f3700 1 -- 172.21.15.59:0/4215767015 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888101fb0 con 0x7fa8881039c0 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.039+0000 7fa88e0f3700 1 --2- 172.21.15.59:0/4215767015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888103dd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa8840098e0 tx=0x7fa884009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5613c8e2818557dd server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.039+0000 7fa88d0f1700 1 -- 172.21.15.59:0/4215767015 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa88400b860 con 0x7fa8881039c0 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.039+0000 7fa88d0f1700 1 -- 172.21.15.59:0/4215767015 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa88400be70 con 0x7fa8881039c0 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.039+0000 7fa88d0f1700 1 -- 172.21.15.59:0/4215767015 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa884010380 con 0x7fa8881039c0 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.039+0000 7fa890357700 1 -- 172.21.15.59:0/4215767015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 msgr2=0x7fa888103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.039+0000 7fa890357700 1 --2- 172.21.15.59:0/4215767015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888103dd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa8840098e0 tx=0x7fa884009250 comp rx=0 tx=0).stop 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.040+0000 7fa890357700 1 -- 172.21.15.59:0/4215767015 shutdown_connections 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.040+0000 7fa890357700 1 --2- 172.21.15.59:0/4215767015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888103dd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.040+0000 7fa890357700 1 -- 172.21.15.59:0/4215767015 >> 172.21.15.59:0/4215767015 conn(0x7fa8880fda40 msgr2=0x7fa8880ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.040+0000 7fa890357700 1 -- 172.21.15.59:0/4215767015 shutdown_connections 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.040+0000 7fa890357700 1 -- 172.21.15.59:0/4215767015 wait complete. 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.040+0000 7fa890357700 1 Processor -- start 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.040+0000 7fa890357700 1 -- start start 2023-12-19T18:39:44.577 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.040+0000 7fa890357700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.040+0000 7fa890357700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa888111670 con 0x7fa8881039c0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.041+0000 7fa88e0f3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.041+0000 7fa88e0f3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45560/0 (socket says 172.21.15.59:45560) 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.041+0000 7fa88e0f3700 1 -- 172.21.15.59:0/3534429927 learned_addr learned my addr 172.21.15.59:0/3534429927 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.041+0000 7fa88e0f3700 1 -- 172.21.15.59:0/3534429927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8840094f0 con 0x7fa8881039c0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.041+0000 7fa88e0f3700 1 --2- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888111130 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa884003080 tx=0x7fa884009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.041+0000 7fa87effd700 1 -- 172.21.15.59:0/3534429927 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa884010800 con 0x7fa8881039c0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.041+0000 7fa87effd700 1 -- 172.21.15.59:0/3534429927 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa884010e10 con 0x7fa8881039c0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.042+0000 7fa87effd700 1 -- 172.21.15.59:0/3534429927 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa88401a430 con 0x7fa8881039c0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.042+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa888111870 con 0x7fa8881039c0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.042+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa888111bc0 con 0x7fa8881039c0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.042+0000 7fa87effd700 1 -- 172.21.15.59:0/3534429927 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fa884024790 con 0x7fa8881039c0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.043+0000 7fa87effd700 1 --2- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa874036640 0x7fa874038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.043+0000 7fa87effd700 1 -- 172.21.15.59:0/3534429927 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa88404a5b0 con 0x7fa8881039c0 2023-12-19T18:39:44.578 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.043+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88805f270 con 0x7fa8881039c0 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.043+0000 7fa88d8f2700 1 --2- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa874036640 0x7fa874038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.043+0000 7fa88d8f2700 1 --2- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa874036640 0x7fa874038af0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa878009990 tx=0x7fa878006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.048+0000 7fa87effd700 1 -- 172.21.15.59:0/3534429927 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa884022990 con 0x7fa8881039c0 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.270+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa88810ada0 con 0x7fa874036640 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.274+0000 7fa87effd700 1 -- 172.21.15.59:0/3534429927 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa88810ada0 con 0x7fa874036640 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa874036640 msgr2=0x7fa874038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 --2- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa874036640 0x7fa874038af0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa878009990 tx=0x7fa878006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 msgr2=0x7fa888111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 --2- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888111130 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa884003080 tx=0x7fa884009250 comp rx=0 tx=0).stop 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 shutdown_connections 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 --2- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa874036640 0x7fa874038af0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 --2- 172.21.15.59:0/3534429927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8881039c0 0x7fa888111130 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 >> 172.21.15.59:0/3534429927 conn(0x7fa8880fda40 msgr2=0x7fa8880ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:44.579 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 shutdown_connections 2023-12-19T18:39:44.580 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:43.276+0000 7fa890357700 1 -- 172.21.15.59:0/3534429927 wait complete. 2023-12-19T18:39:44.580 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying grafana service with default placement... 2023-12-19T18:39:45.686 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2023-12-19T18:39:45.686 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.972+0000 7f7f320fe700 1 Processor -- start 2023-12-19T18:39:45.686 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.972+0000 7f7f320fe700 1 -- start start 2023-12-19T18:39:45.688 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.972+0000 7f7f320fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:45.688 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.972+0000 7f7f320fe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c101b20 con 0x7f7f2c1011d0 2023-12-19T18:39:45.688 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.973+0000 7f7f2b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:45.688 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.973+0000 7f7f2b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45568/0 (socket says 172.21.15.59:45568) 2023-12-19T18:39:45.688 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.973+0000 7f7f2b7fe700 1 -- 172.21.15.59:0/1552218434 learned_addr learned my addr 172.21.15.59:0/1552218434 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:45.688 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.973+0000 7f7f2b7fe700 1 -- 172.21.15.59:0/1552218434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c101c60 con 0x7f7f2c1011d0 2023-12-19T18:39:45.689 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.973+0000 7f7f2b7fe700 1 --2- 172.21.15.59:0/1552218434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1015e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009e50 tx=0x7f7f1c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793c0567c21f8caa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:45.689 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.973+0000 7f7f2a7fc700 1 -- 172.21.15.59:0/1552218434 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f1c0092e0 con 0x7f7f2c1011d0 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.973+0000 7f7f2a7fc700 1 -- 172.21.15.59:0/1552218434 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f1c012430 con 0x7f7f2c1011d0 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.973+0000 7f7f2a7fc700 1 -- 172.21.15.59:0/1552218434 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f1c0075c0 con 0x7f7f2c1011d0 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.974+0000 7f7f320fe700 1 -- 172.21.15.59:0/1552218434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 msgr2=0x7f7f2c1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.974+0000 7f7f320fe700 1 --2- 172.21.15.59:0/1552218434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1015e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009e50 tx=0x7f7f1c009840 comp rx=0 tx=0).stop 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.974+0000 7f7f320fe700 1 -- 172.21.15.59:0/1552218434 shutdown_connections 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.974+0000 7f7f320fe700 1 --2- 172.21.15.59:0/1552218434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1015e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.974+0000 7f7f320fe700 1 -- 172.21.15.59:0/1552218434 >> 172.21.15.59:0/1552218434 conn(0x7f7f2c0fc970 msgr2=0x7f7f2c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.974+0000 7f7f320fe700 1 -- 172.21.15.59:0/1552218434 shutdown_connections 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.974+0000 7f7f320fe700 1 -- 172.21.15.59:0/1552218434 wait complete. 2023-12-19T18:39:45.694 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.975+0000 7f7f320fe700 1 Processor -- start 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.975+0000 7f7f320fe700 1 -- start start 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.975+0000 7f7f320fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1111e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.975+0000 7f7f320fe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c111720 con 0x7f7f2c1011d0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.975+0000 7f7f2b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1111e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.975+0000 7f7f2b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1111e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45574/0 (socket says 172.21.15.59:45574) 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.975+0000 7f7f2b7fe700 1 -- 172.21.15.59:0/823746164 learned_addr learned my addr 172.21.15.59:0/823746164 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.976+0000 7f7f2b7fe700 1 -- 172.21.15.59:0/823746164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c009ae0 con 0x7f7f2c1011d0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.976+0000 7f7f2b7fe700 1 --2- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1111e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009fd0 tx=0x7f7f1c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.976+0000 7f7f13fff700 1 -- 172.21.15.59:0/823746164 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f1c0092e0 con 0x7f7f2c1011d0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.976+0000 7f7f13fff700 1 -- 172.21.15.59:0/823746164 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f1c012920 con 0x7f7f2c1011d0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.976+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c111920 con 0x7f7f2c1011d0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.976+0000 7f7f13fff700 1 -- 172.21.15.59:0/823746164 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f1c0258e0 con 0x7f7f2c1011d0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.976+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c111d30 con 0x7f7f2c1011d0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.977+0000 7f7f13fff700 1 -- 172.21.15.59:0/823746164 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f7f1c019050 con 0x7f7f2c1011d0 2023-12-19T18:39:45.695 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.977+0000 7f7f13fff700 1 --2- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f7f14036640 0x7f7f14038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.977+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c04bb80 con 0x7f7f2c1011d0 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.977+0000 7f7f13fff700 1 -- 172.21.15.59:0/823746164 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7f1c049df0 con 0x7f7f2c1011d0 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.977+0000 7f7f2affd700 1 --2- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f7f14036640 0x7f7f14038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.977+0000 7f7f2affd700 1 --2- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f7f14036640 0x7f7f14038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009990 tx=0x7f7f20006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:44.982+0000 7f7f13fff700 1 -- 172.21.15.59:0/823746164 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7f1c020080 con 0x7f7f2c1011d0 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.189+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f2c0fec40 con 0x7f7f14036640 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.194+0000 7f7f13fff700 1 -- 172.21.15.59:0/823746164 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f7f2c0fec40 con 0x7f7f14036640 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f7f14036640 msgr2=0x7f7f14038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 --2- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f7f14036640 0x7f7f14038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009990 tx=0x7f7f20006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 msgr2=0x7f7f2c1111e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 --2- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1111e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009fd0 tx=0x7f7f1c009840 comp rx=0 tx=0).stop 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 shutdown_connections 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 --2- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f7f14036640 0x7f7f14038af0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 --2- 172.21.15.59:0/823746164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f2c1011d0 0x7f7f2c1111e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:45.696 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 >> 172.21.15.59:0/823746164 conn(0x7f7f2c0fc970 msgr2=0x7f7f2c070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:45.697 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 shutdown_connections 2023-12-19T18:39:45.697 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:45.196+0000 7f7f320fe700 1 -- 172.21.15.59:0/823746164 wait complete. 2023-12-19T18:39:45.697 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying node-exporter service with default placement... 2023-12-19T18:39:46.447 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:46 smithi059 ceph-mon[80661]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:46.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:46 smithi059 ceph-mon[80661]: Saving service grafana spec with placement count:1 2023-12-19T18:39:46.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:46 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:46.888 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2023-12-19T18:39:46.889 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.077+0000 7f93dfc7d700 1 Processor -- start 2023-12-19T18:39:46.889 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.078+0000 7f93dfc7d700 1 -- start start 2023-12-19T18:39:46.889 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.078+0000 7f93dfc7d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:46.889 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.078+0000 7f93dfc7d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d8104310 con 0x7f93d81039c0 2023-12-19T18:39:46.889 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.078+0000 7f93dda19700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.078+0000 7f93dda19700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45576/0 (socket says 172.21.15.59:45576) 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.078+0000 7f93dda19700 1 -- 172.21.15.59:0/351973835 learned_addr learned my addr 172.21.15.59:0/351973835 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.078+0000 7f93dda19700 1 -- 172.21.15.59:0/351973835 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d8101fb0 con 0x7f93d81039c0 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.078+0000 7f93dda19700 1 --2- 172.21.15.59:0/351973835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8103dd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f93d40098e0 tx=0x7f93d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a2a26dfa24a96c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dca17700 1 -- 172.21.15.59:0/351973835 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93d400b860 con 0x7f93d81039c0 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dca17700 1 -- 172.21.15.59:0/351973835 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93d400be70 con 0x7f93d81039c0 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dca17700 1 -- 172.21.15.59:0/351973835 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93d4010380 con 0x7f93d81039c0 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dfc7d700 1 -- 172.21.15.59:0/351973835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 msgr2=0x7f93d8103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dfc7d700 1 --2- 172.21.15.59:0/351973835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8103dd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f93d40098e0 tx=0x7f93d4009250 comp rx=0 tx=0).stop 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dfc7d700 1 -- 172.21.15.59:0/351973835 shutdown_connections 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dfc7d700 1 --2- 172.21.15.59:0/351973835 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8103dd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dfc7d700 1 -- 172.21.15.59:0/351973835 >> 172.21.15.59:0/351973835 conn(0x7f93d80fda40 msgr2=0x7f93d80ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dfc7d700 1 -- 172.21.15.59:0/351973835 shutdown_connections 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.079+0000 7f93dfc7d700 1 -- 172.21.15.59:0/351973835 wait complete. 2023-12-19T18:39:46.890 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.080+0000 7f93dfc7d700 1 Processor -- start 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.080+0000 7f93dfc7d700 1 -- start start 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.080+0000 7f93dfc7d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.080+0000 7f93dfc7d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d8111670 con 0x7f93d81039c0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.080+0000 7f93dda19700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.080+0000 7f93dda19700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45588/0 (socket says 172.21.15.59:45588) 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.080+0000 7f93dda19700 1 -- 172.21.15.59:0/523537498 learned_addr learned my addr 172.21.15.59:0/523537498 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.080+0000 7f93dda19700 1 -- 172.21.15.59:0/523537498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d40094f0 con 0x7f93d81039c0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.080+0000 7f93dda19700 1 --2- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8111130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f93d400bac0 tx=0x7f93d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.081+0000 7f93ce7fc700 1 -- 172.21.15.59:0/523537498 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93d40109b0 con 0x7f93d81039c0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.081+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d8111870 con 0x7f93d81039c0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.081+0000 7f93ce7fc700 1 -- 172.21.15.59:0/523537498 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93d401a430 con 0x7f93d81039c0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.081+0000 7f93ce7fc700 1 -- 172.21.15.59:0/523537498 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93d40245b0 con 0x7f93d81039c0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.081+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d8111c80 con 0x7f93d81039c0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.081+0000 7f93ce7fc700 1 -- 172.21.15.59:0/523537498 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f93d4024820 con 0x7f93d81039c0 2023-12-19T18:39:46.891 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.082+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d8102d30 con 0x7f93d81039c0 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.082+0000 7f93ce7fc700 1 --2- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f93c40365f0 0x7f93c4038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.082+0000 7f93ce7fc700 1 -- 172.21.15.59:0/523537498 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f93d404a6e0 con 0x7f93d81039c0 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.082+0000 7f93dd218700 1 --2- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f93c40365f0 0x7f93c4038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.082+0000 7f93dd218700 1 --2- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f93c40365f0 0x7f93c4038aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009990 tx=0x7f93c8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.086+0000 7f93ce7fc700 1 -- 172.21.15.59:0/523537498 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f93d4010b10 con 0x7f93d81039c0 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.295+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f93d810ae30 con 0x7f93c40365f0 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.299+0000 7f93ce7fc700 1 -- 172.21.15.59:0/523537498 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f93d810ae30 con 0x7f93c40365f0 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f93c40365f0 msgr2=0x7f93c4038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 --2- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f93c40365f0 0x7f93c4038aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009990 tx=0x7f93c8006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 msgr2=0x7f93d8111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 --2- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8111130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f93d400bac0 tx=0x7f93d4009250 comp rx=0 tx=0).stop 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 shutdown_connections 2023-12-19T18:39:46.892 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 --2- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f93c40365f0 0x7f93c4038aa0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:46.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 --2- 172.21.15.59:0/523537498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93d81039c0 0x7f93d8111130 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:46.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 >> 172.21.15.59:0/523537498 conn(0x7f93d80fda40 msgr2=0x7f93d80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:46.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 shutdown_connections 2023-12-19T18:39:46.893 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:46.301+0000 7f93dfc7d700 1 -- 172.21.15.59:0/523537498 wait complete. 2023-12-19T18:39:46.893 INFO:teuthology.orchestra.run.smithi059.stdout:Deploying alertmanager service with default placement... 2023-12-19T18:39:47.483 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:47 smithi059 ceph-mon[80661]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:47.483 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:47 smithi059 ceph-mon[80661]: Saving service node-exporter spec with placement * 2023-12-19T18:39:47.483 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:47 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:48.065 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2023-12-19T18:39:48.065 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.253+0000 7fb41294f700 1 Processor -- start 2023-12-19T18:39:48.065 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.253+0000 7fb41294f700 1 -- start start 2023-12-19T18:39:48.065 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.253+0000 7fb41294f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:48.066 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.253+0000 7fb41294f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c104310 con 0x7fb40c1039c0 2023-12-19T18:39:48.066 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.253+0000 7fb40bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:48.066 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.253+0000 7fb40bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45598/0 (socket says 172.21.15.59:45598) 2023-12-19T18:39:48.066 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.253+0000 7fb40bfff700 1 -- 172.21.15.59:0/1120438309 learned_addr learned my addr 172.21.15.59:0/1120438309 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:48.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.253+0000 7fb40bfff700 1 -- 172.21.15.59:0/1120438309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c101fb0 con 0x7fb40c1039c0 2023-12-19T18:39:48.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.253+0000 7fb40bfff700 1 --2- 172.21.15.59:0/1120438309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c103dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0098e0 tx=0x7fb3fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ff981a54bafe71 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:48.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb40affd700 1 -- 172.21.15.59:0/1120438309 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3fc00b860 con 0x7fb40c1039c0 2023-12-19T18:39:48.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb40affd700 1 -- 172.21.15.59:0/1120438309 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3fc00be70 con 0x7fb40c1039c0 2023-12-19T18:39:48.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb40affd700 1 -- 172.21.15.59:0/1120438309 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3fc010380 con 0x7fb40c1039c0 2023-12-19T18:39:48.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb41294f700 1 -- 172.21.15.59:0/1120438309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 msgr2=0x7fb40c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:48.070 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb41294f700 1 --2- 172.21.15.59:0/1120438309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c103dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0098e0 tx=0x7fb3fc009250 comp rx=0 tx=0).stop 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb41294f700 1 -- 172.21.15.59:0/1120438309 shutdown_connections 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb41294f700 1 --2- 172.21.15.59:0/1120438309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c103dd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb41294f700 1 -- 172.21.15.59:0/1120438309 >> 172.21.15.59:0/1120438309 conn(0x7fb40c0fda40 msgr2=0x7fb40c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb41294f700 1 -- 172.21.15.59:0/1120438309 shutdown_connections 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.254+0000 7fb41294f700 1 -- 172.21.15.59:0/1120438309 wait complete. 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.255+0000 7fb41294f700 1 Processor -- start 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.255+0000 7fb41294f700 1 -- start start 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.255+0000 7fb41294f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.255+0000 7fb41294f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c077190 con 0x7fb40c1039c0 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.255+0000 7fb40bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.255+0000 7fb40bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45604/0 (socket says 172.21.15.59:45604) 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.255+0000 7fb40bfff700 1 -- 172.21.15.59:0/1966128611 learned_addr learned my addr 172.21.15.59:0/1966128611 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.256+0000 7fb40bfff700 1 -- 172.21.15.59:0/1966128611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc0094f0 con 0x7fb40c1039c0 2023-12-19T18:39:48.071 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.256+0000 7fb40bfff700 1 --2- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c078d10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00bac0 tx=0x7fb3fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.256+0000 7fb408ff9700 1 -- 172.21.15.59:0/1966128611 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3fc0109b0 con 0x7fb40c1039c0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.256+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c077390 con 0x7fb40c1039c0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.256+0000 7fb408ff9700 1 -- 172.21.15.59:0/1966128611 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3fc01a430 con 0x7fb40c1039c0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.256+0000 7fb408ff9700 1 -- 172.21.15.59:0/1966128611 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3fc0245b0 con 0x7fb40c1039c0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.256+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c0777a0 con 0x7fb40c1039c0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.257+0000 7fb408ff9700 1 -- 172.21.15.59:0/1966128611 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fb3fc024820 con 0x7fb40c1039c0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.257+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40c10b190 con 0x7fb40c1039c0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.257+0000 7fb408ff9700 1 --2- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb3f40365f0 0x7fb3f4038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.257+0000 7fb408ff9700 1 -- 172.21.15.59:0/1966128611 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb3fc04a580 con 0x7fb40c1039c0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.257+0000 7fb40b7fe700 1 --2- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb3f40365f0 0x7fb3f4038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.258+0000 7fb40b7fe700 1 --2- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb3f40365f0 0x7fb3f4038aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb400009990 tx=0x7fb400006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.262+0000 7fb408ff9700 1 -- 172.21.15.59:0/1966128611 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb3fc018940 con 0x7fb40c1039c0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.474+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb40c077aa0 con 0x7fb3f40365f0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.481+0000 7fb408ff9700 1 -- 172.21.15.59:0/1966128611 <== mgr.14118 v2:172.21.15.59:6800/2067989811 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb40c077aa0 con 0x7fb3f40365f0 2023-12-19T18:39:48.072 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.483+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb3f40365f0 msgr2=0x7fb3f4038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:48.073 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.483+0000 7fb41294f700 1 --2- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb3f40365f0 0x7fb3f4038aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb400009990 tx=0x7fb400006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:48.073 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.483+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 msgr2=0x7fb40c078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:48.073 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.483+0000 7fb41294f700 1 --2- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c078d10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00bac0 tx=0x7fb3fc009250 comp rx=0 tx=0).stop 2023-12-19T18:39:48.073 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.483+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 shutdown_connections 2023-12-19T18:39:48.073 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.483+0000 7fb41294f700 1 --2- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb3f40365f0 0x7fb3f4038aa0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:48.073 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.483+0000 7fb41294f700 1 --2- 172.21.15.59:0/1966128611 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb40c1039c0 0x7fb40c078d10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:48.073 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.483+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 >> 172.21.15.59:0/1966128611 conn(0x7fb40c0fda40 msgr2=0x7fb40c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:48.073 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.483+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 shutdown_connections 2023-12-19T18:39:48.073 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:47.484+0000 7fb41294f700 1 -- 172.21.15.59:0/1966128611 wait complete. 2023-12-19T18:39:48.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:48 smithi059 ceph-mon[80661]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:39:48.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:48 smithi059 ceph-mon[80661]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:39:48.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:48 smithi059 ceph-mon[80661]: Saving service alertmanager spec with placement count:1 2023-12-19T18:39:48.664 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:48 smithi059 ceph-mon[80661]: from='mgr.14118 172.21.15.59:0/1203771061' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:49.176 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.459+0000 7fb028501700 1 Processor -- start 2023-12-19T18:39:49.176 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.459+0000 7fb028501700 1 -- start start 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.459+0000 7fb028501700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.459+0000 7fb028501700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb020104310 con 0x7fb0201039c0 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.459+0000 7fb02629d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.459+0000 7fb02629d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45614/0 (socket says 172.21.15.59:45614) 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.459+0000 7fb02629d700 1 -- 172.21.15.59:0/548080121 learned_addr learned my addr 172.21.15.59:0/548080121 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.459+0000 7fb02629d700 1 -- 172.21.15.59:0/548080121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb020101fb0 con 0x7fb0201039c0 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.460+0000 7fb02629d700 1 --2- 172.21.15.59:0/548080121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020103dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0098e0 tx=0x7fb01c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3ab7a6a8f5a710 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.460+0000 7fb02529b700 1 -- 172.21.15.59:0/548080121 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb01c00b860 con 0x7fb0201039c0 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.460+0000 7fb02529b700 1 -- 172.21.15.59:0/548080121 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb01c00be70 con 0x7fb0201039c0 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.460+0000 7fb02529b700 1 -- 172.21.15.59:0/548080121 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb01c010380 con 0x7fb0201039c0 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.461+0000 7fb028501700 1 -- 172.21.15.59:0/548080121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 msgr2=0x7fb020103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:49.177 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.461+0000 7fb028501700 1 --2- 172.21.15.59:0/548080121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020103dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0098e0 tx=0x7fb01c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.461+0000 7fb028501700 1 -- 172.21.15.59:0/548080121 shutdown_connections 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.461+0000 7fb028501700 1 --2- 172.21.15.59:0/548080121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020103dd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.461+0000 7fb028501700 1 -- 172.21.15.59:0/548080121 >> 172.21.15.59:0/548080121 conn(0x7fb0200fda40 msgr2=0x7fb0200ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.461+0000 7fb028501700 1 -- 172.21.15.59:0/548080121 shutdown_connections 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.461+0000 7fb028501700 1 -- 172.21.15.59:0/548080121 wait complete. 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.461+0000 7fb028501700 1 Processor -- start 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.461+0000 7fb028501700 1 -- start start 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb028501700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb028501700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb020077190 con 0x7fb0201039c0 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb02629d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb02629d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45626/0 (socket says 172.21.15.59:45626) 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb02629d700 1 -- 172.21.15.59:0/2325333884 learned_addr learned my addr 172.21.15.59:0/2325333884 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb02629d700 1 -- 172.21.15.59:0/2325333884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0094f0 con 0x7fb0201039c0 2023-12-19T18:39:49.178 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb02629d700 1 --2- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020078d10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb01c003080 tx=0x7fb01c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:49.179 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb016ffd700 1 -- 172.21.15.59:0/2325333884 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb01c010800 con 0x7fb0201039c0 2023-12-19T18:39:49.179 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb016ffd700 1 -- 172.21.15.59:0/2325333884 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb01c010e10 con 0x7fb0201039c0 2023-12-19T18:39:49.179 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb020077390 con 0x7fb0201039c0 2023-12-19T18:39:49.180 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb016ffd700 1 -- 172.21.15.59:0/2325333884 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb01c01a430 con 0x7fb0201039c0 2023-12-19T18:39:49.180 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.462+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0200777a0 con 0x7fb0201039c0 2023-12-19T18:39:49.180 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.463+0000 7fb016ffd700 1 -- 172.21.15.59:0/2325333884 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fb01c024790 con 0x7fb0201039c0 2023-12-19T18:39:49.180 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.463+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02005f270 con 0x7fb0201039c0 2023-12-19T18:39:49.180 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.463+0000 7fb016ffd700 1 --2- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb00c036640 0x7fb00c038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:49.180 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.463+0000 7fb016ffd700 1 -- 172.21.15.59:0/2325333884 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb01c04a5e0 con 0x7fb0201039c0 2023-12-19T18:39:49.180 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.464+0000 7fb025a9c700 1 --2- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb00c036640 0x7fb00c038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:49.180 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.464+0000 7fb025a9c700 1 --2- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb00c036640 0x7fb00c038af0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb010009990 tx=0x7fb010006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:49.180 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.468+0000 7fb016ffd700 1 -- 172.21.15.59:0/2325333884 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb01c024440 con 0x7fb0201039c0 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.664+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb020077c20 con 0x7fb0201039c0 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.668+0000 7fb016ffd700 1 -- 172.21.15.59:0/2325333884 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb01c022340 con 0x7fb0201039c0 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb00c036640 msgr2=0x7fb00c038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 --2- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb00c036640 0x7fb00c038af0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb010009990 tx=0x7fb010006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 msgr2=0x7fb020078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 --2- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020078d10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb01c003080 tx=0x7fb01c009250 comp rx=0 tx=0).stop 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 shutdown_connections 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 --2- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fb00c036640 0x7fb00c038af0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 --2- 172.21.15.59:0/2325333884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0201039c0 0x7fb020078d10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 >> 172.21.15.59:0/2325333884 conn(0x7fb0200fda40 msgr2=0x7fb0200ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 shutdown_connections 2023-12-19T18:39:49.181 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:48.670+0000 7fb028501700 1 -- 172.21.15.59:0/2325333884 wait complete. 2023-12-19T18:39:49.808 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:49 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2325333884' entity='client.admin' 2023-12-19T18:39:49.808 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:49 smithi059 ceph-mon[80661]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:39:50.454 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.599+0000 7f68661f3700 1 Processor -- start 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.600+0000 7f68661f3700 1 -- start start 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.600+0000 7f68661f3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f6860104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.600+0000 7f68661f3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6860105420 con 0x7f6860070fc0 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.600+0000 7f685f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f6860104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.600+0000 7f685f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f6860104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45628/0 (socket says 172.21.15.59:45628) 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.600+0000 7f685f7fe700 1 -- 172.21.15.59:0/4069288618 learned_addr learned my addr 172.21.15.59:0/4069288618 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.600+0000 7f685f7fe700 1 -- 172.21.15.59:0/4069288618 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6860105560 con 0x7f6860070fc0 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.601+0000 7f685f7fe700 1 --2- 172.21.15.59:0/4069288618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f6860104ee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f68500098e0 tx=0x7f6850009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbddc7966b41169c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.601+0000 7f685e7fc700 1 -- 172.21.15.59:0/4069288618 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f685000b860 con 0x7f6860070fc0 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.601+0000 7f685e7fc700 1 -- 172.21.15.59:0/4069288618 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f685000be70 con 0x7f6860070fc0 2023-12-19T18:39:50.460 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.601+0000 7f685e7fc700 1 -- 172.21.15.59:0/4069288618 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6850010380 con 0x7f6860070fc0 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.602+0000 7f68661f3700 1 -- 172.21.15.59:0/4069288618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 msgr2=0x7f6860104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.602+0000 7f68661f3700 1 --2- 172.21.15.59:0/4069288618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f6860104ee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f68500098e0 tx=0x7f6850009250 comp rx=0 tx=0).stop 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.602+0000 7f68661f3700 1 -- 172.21.15.59:0/4069288618 shutdown_connections 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.602+0000 7f68661f3700 1 --2- 172.21.15.59:0/4069288618 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f6860104ee0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.602+0000 7f68661f3700 1 -- 172.21.15.59:0/4069288618 >> 172.21.15.59:0/4069288618 conn(0x7f68600fc970 msgr2=0x7f68600fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.602+0000 7f68661f3700 1 -- 172.21.15.59:0/4069288618 shutdown_connections 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.602+0000 7f68661f3700 1 -- 172.21.15.59:0/4069288618 wait complete. 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.602+0000 7f68661f3700 1 Processor -- start 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f68661f3700 1 -- start start 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f68661f3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f68601111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f68661f3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6860111700 con 0x7f6860070fc0 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f685f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f68601111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:50.461 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f685f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f68601111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45640/0 (socket says 172.21.15.59:45640) 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f685f7fe700 1 -- 172.21.15.59:0/566138784 learned_addr learned my addr 172.21.15.59:0/566138784 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f685f7fe700 1 -- 172.21.15.59:0/566138784 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68500094f0 con 0x7f6860070fc0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f685f7fe700 1 --2- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f68601111c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f68500094c0 tx=0x7f6850009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f6847fff700 1 -- 172.21.15.59:0/566138784 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6850010850 con 0x7f6860070fc0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f6847fff700 1 -- 172.21.15.59:0/566138784 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6850010e60 con 0x7f6860070fc0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6860111900 con 0x7f6860070fc0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.604+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6860111d10 con 0x7f6860070fc0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.603+0000 7f6847fff700 1 -- 172.21.15.59:0/566138784 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f685001a430 con 0x7f6860070fc0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.604+0000 7f6847fff700 1 -- 172.21.15.59:0/566138784 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f68500109b0 con 0x7f6860070fc0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.605+0000 7f6847fff700 1 --2- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f6848036640 0x7f6848038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.605+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686004bb80 con 0x7f6860070fc0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.605+0000 7f6847fff700 1 -- 172.21.15.59:0/566138784 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f685004a3f0 con 0x7f6860070fc0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.605+0000 7f685effd700 1 --2- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f6848036640 0x7f6848038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.605+0000 7f685effd700 1 --2- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f6848036640 0x7f6848038af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6854009990 tx=0x7f6854006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:50.462 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.609+0000 7f6847fff700 1 -- 172.21.15.59:0/566138784 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6850010c60 con 0x7f6860070fc0 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.808+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f68601062e0 con 0x7f6860070fc0 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.812+0000 7f6847fff700 1 -- 172.21.15.59:0/566138784 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f6850016900 con 0x7f6860070fc0 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.814+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f6848036640 msgr2=0x7f6848038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.814+0000 7f68661f3700 1 --2- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f6848036640 0x7f6848038af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6854009990 tx=0x7f6854006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.814+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 msgr2=0x7f68601111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.814+0000 7f68661f3700 1 --2- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f68601111c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f68500094c0 tx=0x7f6850009250 comp rx=0 tx=0).stop 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.814+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 shutdown_connections 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.814+0000 7f68661f3700 1 --2- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f6848036640 0x7f6848038af0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.814+0000 7f68661f3700 1 --2- 172.21.15.59:0/566138784 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6860070fc0 0x7f68601111c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.814+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 >> 172.21.15.59:0/566138784 conn(0x7f68600fc970 msgr2=0x7f68600fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.815+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 shutdown_connections 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:49.815+0000 7f68661f3700 1 -- 172.21.15.59:0/566138784 wait complete. 2023-12-19T18:39:50.463 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling the dashboard module... 2023-12-19T18:39:50.988 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:50 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/566138784' entity='client.admin' 2023-12-19T18:39:52.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:51 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/4136618842' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-19T18:39:52.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:51 smithi059 ceph-mon[80661]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:39:52.426 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.015+0000 7f301203c700 1 Processor -- start 2023-12-19T18:39:52.426 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.015+0000 7f301203c700 1 -- start start 2023-12-19T18:39:52.426 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.015+0000 7f301203c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:52.426 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.015+0000 7f301203c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c105420 con 0x7f300c076c20 2023-12-19T18:39:52.426 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.015+0000 7f300b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.016+0000 7f300b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45642/0 (socket says 172.21.15.59:45642) 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.016+0000 7f300b7fe700 1 -- 172.21.15.59:0/3223863652 learned_addr learned my addr 172.21.15.59:0/3223863652 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.016+0000 7f300b7fe700 1 -- 172.21.15.59:0/3223863652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c105560 con 0x7f300c076c20 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.016+0000 7f300b7fe700 1 --2- 172.21.15.59:0/3223863652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c104ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0098e0 tx=0x7f2ffc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e6ab77166b982c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.016+0000 7f300a7fc700 1 -- 172.21.15.59:0/3223863652 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ffc00b860 con 0x7f300c076c20 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.016+0000 7f300a7fc700 1 -- 172.21.15.59:0/3223863652 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ffc00be70 con 0x7f300c076c20 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.016+0000 7f300a7fc700 1 -- 172.21.15.59:0/3223863652 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ffc010380 con 0x7f300c076c20 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.017+0000 7f301203c700 1 -- 172.21.15.59:0/3223863652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 msgr2=0x7f300c104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.017+0000 7f301203c700 1 --2- 172.21.15.59:0/3223863652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c104ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0098e0 tx=0x7f2ffc009250 comp rx=0 tx=0).stop 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.017+0000 7f301203c700 1 -- 172.21.15.59:0/3223863652 shutdown_connections 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.017+0000 7f301203c700 1 --2- 172.21.15.59:0/3223863652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c104ee0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.017+0000 7f301203c700 1 -- 172.21.15.59:0/3223863652 >> 172.21.15.59:0/3223863652 conn(0x7f300c0fc950 msgr2=0x7f300c0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.017+0000 7f301203c700 1 -- 172.21.15.59:0/3223863652 shutdown_connections 2023-12-19T18:39:52.427 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.017+0000 7f301203c700 1 -- 172.21.15.59:0/3223863652 wait complete. 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.017+0000 7f301203c700 1 Processor -- start 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.018+0000 7f301203c700 1 -- start start 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.018+0000 7f301203c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c111190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.018+0000 7f301203c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c1116d0 con 0x7f300c076c20 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.018+0000 7f300b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c111190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.018+0000 7f300b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c111190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45648/0 (socket says 172.21.15.59:45648) 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.018+0000 7f300b7fe700 1 -- 172.21.15.59:0/4136618842 learned_addr learned my addr 172.21.15.59:0/4136618842 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.018+0000 7f300b7fe700 1 -- 172.21.15.59:0/4136618842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc0094f0 con 0x7f300c076c20 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.018+0000 7f300b7fe700 1 --2- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c111190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0094c0 tx=0x7f2ffc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.018+0000 7f2ff3fff700 1 -- 172.21.15.59:0/4136618842 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ffc010850 con 0x7f300c076c20 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.019+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c1118d0 con 0x7f300c076c20 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.019+0000 7f2ff3fff700 1 -- 172.21.15.59:0/4136618842 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ffc010e60 con 0x7f300c076c20 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.019+0000 7f2ff3fff700 1 -- 172.21.15.59:0/4136618842 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ffc01a430 con 0x7f300c076c20 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.019+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c111ce0 con 0x7f300c076c20 2023-12-19T18:39:52.428 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.019+0000 7f2ff3fff700 1 -- 172.21.15.59:0/4136618842 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f2ffc01a6b0 con 0x7f300c076c20 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.020+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c04bb80 con 0x7f300c076c20 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.020+0000 7f2ff3fff700 1 --2- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f2ff40365f0 0x7f2ff4038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.020+0000 7f2ff3fff700 1 -- 172.21.15.59:0/4136618842 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2ffc04a400 con 0x7f300c076c20 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.020+0000 7f300affd700 1 --2- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f2ff40365f0 0x7f2ff4038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.020+0000 7f300affd700 1 --2- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f2ff40365f0 0x7f2ff4038aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3000009990 tx=0x7f3000006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.024+0000 7f2ff3fff700 1 -- 172.21.15.59:0/4136618842 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2ffc0189a0 con 0x7f300c076c20 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.273+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f300c10af80 con 0x7f300c076c20 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.816+0000 7f2ff3fff700 1 -- 172.21.15.59:0/4136618842 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f2ffc0247b0 con 0x7f300c076c20 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.818+0000 7f2ff3fff700 1 -- 172.21.15.59:0/4136618842 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f2ffc00f4c0 con 0x7f300c076c20 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.820+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f2ff40365f0 msgr2=0x7f2ff4038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.820+0000 7f301203c700 1 --2- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f2ff40365f0 0x7f2ff4038aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3000009990 tx=0x7f3000006e40 comp rx=0 tx=0).stop 2023-12-19T18:39:52.429 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.820+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 msgr2=0x7f300c111190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:52.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.820+0000 7f301203c700 1 --2- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c111190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0094c0 tx=0x7f2ffc009250 comp rx=0 tx=0).stop 2023-12-19T18:39:52.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.821+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 shutdown_connections 2023-12-19T18:39:52.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.821+0000 7f301203c700 1 --2- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7f2ff40365f0 0x7f2ff4038aa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:52.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.821+0000 7f301203c700 1 --2- 172.21.15.59:0/4136618842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f300c076c20 0x7f300c111190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:52.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.821+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 >> 172.21.15.59:0/4136618842 conn(0x7f300c0fc950 msgr2=0x7f300c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:52.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.821+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 shutdown_connections 2023-12-19T18:39:52.430 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:51.821+0000 7f301203c700 1 -- 172.21.15.59:0/4136618842 wait complete. 2023-12-19T18:39:52.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:52 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/4136618842' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-19T18:39:52.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:52 smithi059 ceph-mon[80661]: mgrmap e9: smithi059.ccbnob(active, since 24s) 2023-12-19T18:39:53.708 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2023-12-19T18:39:53.708 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-12-19T18:39:53.708 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-19T18:39:53.708 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "active_name": "smithi059.ccbnob", 2023-12-19T18:39:53.708 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-12-19T18:39:53.708 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2023-12-19T18:39:53.708 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.845+0000 7ff34493a700 1 Processor -- start 2023-12-19T18:39:53.708 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.845+0000 7ff34493a700 1 -- start start 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.845+0000 7ff34493a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340104ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.845+0000 7ff34493a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff340105430 con 0x7ff340076c00 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.845+0000 7ff33e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340104ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.846+0000 7ff33e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340104ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45212/0 (socket says 172.21.15.59:45212) 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.846+0000 7ff33e59c700 1 -- 172.21.15.59:0/939061183 learned_addr learned my addr 172.21.15.59:0/939061183 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.846+0000 7ff33e59c700 1 -- 172.21.15.59:0/939061183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff340105570 con 0x7ff340076c00 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.846+0000 7ff33e59c700 1 --2- 172.21.15.59:0/939061183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340104ef0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff3340098e0 tx=0x7ff334009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae05f085580a56e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.846+0000 7ff33d59a700 1 -- 172.21.15.59:0/939061183 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff33400b860 con 0x7ff340076c00 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.846+0000 7ff33d59a700 1 -- 172.21.15.59:0/939061183 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff33400be70 con 0x7ff340076c00 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.846+0000 7ff33d59a700 1 -- 172.21.15.59:0/939061183 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff334010380 con 0x7ff340076c00 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.847+0000 7ff34493a700 1 -- 172.21.15.59:0/939061183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 msgr2=0x7ff340104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:53.709 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.847+0000 7ff34493a700 1 --2- 172.21.15.59:0/939061183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340104ef0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff3340098e0 tx=0x7ff334009250 comp rx=0 tx=0).stop 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.847+0000 7ff34493a700 1 -- 172.21.15.59:0/939061183 shutdown_connections 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.847+0000 7ff34493a700 1 --2- 172.21.15.59:0/939061183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340104ef0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.847+0000 7ff34493a700 1 -- 172.21.15.59:0/939061183 >> 172.21.15.59:0/939061183 conn(0x7ff3400fc980 msgr2=0x7ff3400fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.847+0000 7ff34493a700 1 -- 172.21.15.59:0/939061183 shutdown_connections 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.847+0000 7ff34493a700 1 -- 172.21.15.59:0/939061183 wait complete. 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.848+0000 7ff34493a700 1 Processor -- start 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.848+0000 7ff34493a700 1 -- start start 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.848+0000 7ff34493a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340102970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.848+0000 7ff34493a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff340100df0 con 0x7ff340076c00 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.848+0000 7ff33e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340102970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.848+0000 7ff33e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340102970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45224/0 (socket says 172.21.15.59:45224) 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.848+0000 7ff33e59c700 1 -- 172.21.15.59:0/1972882221 learned_addr learned my addr 172.21.15.59:0/1972882221 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.848+0000 7ff33e59c700 1 -- 172.21.15.59:0/1972882221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3340094f0 con 0x7ff340076c00 2023-12-19T18:39:53.710 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.849+0000 7ff33e59c700 1 --2- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340102970 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff334006d40 tx=0x7ff334009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.849+0000 7ff32effd700 1 -- 172.21.15.59:0/1972882221 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff334010960 con 0x7ff340076c00 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.849+0000 7ff32effd700 1 -- 172.21.15.59:0/1972882221 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff33401a430 con 0x7ff340076c00 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.849+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff340100ff0 con 0x7ff340076c00 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.849+0000 7ff32effd700 1 -- 172.21.15.59:0/1972882221 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3340245b0 con 0x7ff340076c00 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.849+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff340101400 con 0x7ff340076c00 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.850+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34010b170 con 0x7ff340076c00 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.850+0000 7ff32effd700 1 -- 172.21.15.59:0/1972882221 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7ff33401a9f0 con 0x7ff340076c00 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.850+0000 7ff32effd700 1 --2- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff328036690 0x7ff328038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.851+0000 7ff32effd700 1 -- 172.21.15.59:0/1972882221 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff33404a450 con 0x7ff340076c00 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.851+0000 7ff33dd9b700 1 -- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff328036690 msgr2=0x7ff328038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2067989811 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.851+0000 7ff33dd9b700 1 --2- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff328036690 0x7ff328038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:52.855+0000 7ff32effd700 1 -- 172.21.15.59:0/1972882221 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff334018db0 con 0x7ff340076c00 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.052+0000 7ff33dd9b700 1 -- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff328036690 msgr2=0x7ff328038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2067989811 2023-12-19T18:39:53.711 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.052+0000 7ff33dd9b700 1 --2- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff328036690 0x7ff328038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.108+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff34002b0e0 con 0x7ff340076c00 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.109+0000 7ff32effd700 1 -- 172.21.15.59:0/1972882221 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7ff334018350 con 0x7ff340076c00 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff328036690 msgr2=0x7ff328038b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 --2- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff328036690 0x7ff328038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 msgr2=0x7ff340102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 --2- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340102970 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff334006d40 tx=0x7ff334009250 comp rx=0 tx=0).stop 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 shutdown_connections 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 --2- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7ff328036690 0x7ff328038b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 --2- 172.21.15.59:0/1972882221 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340076c00 0x7ff340102970 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 >> 172.21.15.59:0/1972882221 conn(0x7ff3400fc980 msgr2=0x7ff3400fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 shutdown_connections 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:53.111+0000 7ff34493a700 1 -- 172.21.15.59:0/1972882221 wait complete. 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for the mgr to restart... 2023-12-19T18:39:53.712 INFO:teuthology.orchestra.run.smithi059.stdout:Waiting for mgr epoch 9... 2023-12-19T18:39:54.029 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:53 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1972882221' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-19T18:39:58.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: Active manager daemon smithi059.ccbnob restarted 2023-12-19T18:39:58.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: Activating manager daemon smithi059.ccbnob 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: osdmap e3: 0 total, 0 up, 0 in 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: mgrmap e10: smithi059.ccbnob(active, starting, since 0.00565829s) 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr metadata", "who": "smithi059.ccbnob", "id": "smithi059.ccbnob"}]: dispatch 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: Manager daemon smithi059.ccbnob is now available 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:39:58.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:58 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:39:59.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:59 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/mirror_snapshot_schedule"}]: dispatch 2023-12-19T18:39:59.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:59 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/trash_purge_schedule"}]: dispatch 2023-12-19T18:39:59.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:59 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:39:59.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:39:59 smithi059 ceph-mon[80661]: mgrmap e11: smithi059.ccbnob(active, since 1.00554s) 2023-12-19T18:40:00.195 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout { 2023-12-19T18:40:00.195 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-12-19T18:40:00.195 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout "initialized": true 2023-12-19T18:40:00.195 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout } 2023-12-19T18:40:00.195 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.154+0000 7fa7535cd700 1 Processor -- start 2023-12-19T18:40:00.195 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.154+0000 7fa7535cd700 1 -- start start 2023-12-19T18:40:00.195 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.154+0000 7fa7535cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:00.195 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.154+0000 7fa7535cd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c103100 con 0x7fa74c1027b0 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.155+0000 7fa751369700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.155+0000 7fa751369700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45236/0 (socket says 172.21.15.59:45236) 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.155+0000 7fa751369700 1 -- 172.21.15.59:0/3614119074 learned_addr learned my addr 172.21.15.59:0/3614119074 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.155+0000 7fa751369700 1 -- 172.21.15.59:0/3614119074 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c103240 con 0x7fa74c1027b0 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.155+0000 7fa751369700 1 --2- 172.21.15.59:0/3614119074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c102bc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7480098e0 tx=0x7fa748009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7086b2217717a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.155+0000 7fa73ffff700 1 -- 172.21.15.59:0/3614119074 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa74800b860 con 0x7fa74c1027b0 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.155+0000 7fa73ffff700 1 -- 172.21.15.59:0/3614119074 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa74800be70 con 0x7fa74c1027b0 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.155+0000 7fa73ffff700 1 -- 172.21.15.59:0/3614119074 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa748010380 con 0x7fa74c1027b0 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.156+0000 7fa7535cd700 1 -- 172.21.15.59:0/3614119074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 msgr2=0x7fa74c102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:00.196 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.156+0000 7fa7535cd700 1 --2- 172.21.15.59:0/3614119074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c102bc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7480098e0 tx=0x7fa748009250 comp rx=0 tx=0).stop 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.156+0000 7fa7535cd700 1 -- 172.21.15.59:0/3614119074 shutdown_connections 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.156+0000 7fa7535cd700 1 --2- 172.21.15.59:0/3614119074 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c102bc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.156+0000 7fa7535cd700 1 -- 172.21.15.59:0/3614119074 >> 172.21.15.59:0/3614119074 conn(0x7fa74c0fc0b0 msgr2=0x7fa74c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.156+0000 7fa7535cd700 1 -- 172.21.15.59:0/3614119074 shutdown_connections 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.157+0000 7fa7535cd700 1 -- 172.21.15.59:0/3614119074 wait complete. 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.157+0000 7fa7535cd700 1 Processor -- start 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.157+0000 7fa7535cd700 1 -- start start 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.157+0000 7fa7535cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c113ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.157+0000 7fa7535cd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c1010b0 con 0x7fa74c1027b0 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa751369700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c113ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:00.197 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa751369700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c113ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45238/0 (socket says 172.21.15.59:45238) 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa751369700 1 -- 172.21.15.59:0/3827912418 learned_addr learned my addr 172.21.15.59:0/3827912418 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa751369700 1 -- 172.21.15.59:0/3827912418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7480094f0 con 0x7fa74c1027b0 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa751369700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c113ea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa748011b60 tx=0x7fa748009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa748010850 con 0x7fa74c1027b0 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa748010e60 con 0x7fa74c1027b0 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa7535cd700 1 -- 172.21.15.59:0/3827912418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c1012b0 con 0x7fa74c1027b0 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa74801ae20 con 0x7fa74c1027b0 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.158+0000 7fa7535cd700 1 -- 172.21.15.59:0/3827912418 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c1145a0 con 0x7fa74c1027b0 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.159+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fa74802d3f0 con 0x7fa74c1027b0 2023-12-19T18:40:00.198 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.159+0000 7fa73dffb700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 0x7fa738038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.159+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 --> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa738039200 con 0x7fa738036640 2023-12-19T18:40:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.160+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa74804a540 con 0x7fa74c1027b0 2023-12-19T18:40:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.160+0000 7fa750b68700 1 -- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 msgr2=0x7fa738038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2067989811 2023-12-19T18:40:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.160+0000 7fa750b68700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 0x7fa738038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T18:40:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.361+0000 7fa750b68700 1 -- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 msgr2=0x7fa738038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2067989811 2023-12-19T18:40:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.361+0000 7fa750b68700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 0x7fa738038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T18:40:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.762+0000 7fa750b68700 1 -- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 msgr2=0x7fa738038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2067989811 2023-12-19T18:40:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:54.762+0000 7fa750b68700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 0x7fa738038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T18:40:00.199 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:55.563+0000 7fa750b68700 1 -- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 msgr2=0x7fa738038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2067989811 2023-12-19T18:40:00.200 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:55.563+0000 7fa750b68700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 0x7fa738038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T18:40:00.200 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:57.164+0000 7fa750b68700 1 -- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 msgr2=0x7fa738038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/2067989811 2023-12-19T18:40:00.200 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:57.164+0000 7fa750b68700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 0x7fa738038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T18:40:00.200 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:58.647+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 10) v1 ==== 43061+0+0 (secure 0 0 0) 0x7fa74802d740 con 0x7fa74c1027b0 2023-12-19T18:40:00.200 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:58.647+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 msgr2=0x7fa738038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T18:40:00.200 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:58.647+0000 7fa73dffb700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 0x7fa738038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:00.200 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.646+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7fa74804b810 con 0x7fa74c1027b0 2023-12-19T18:40:00.203 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.646+0000 7fa73dffb700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa738039fb0 0x7fa73803c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:00.203 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.646+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 --> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa738039200 con 0x7fa738039fb0 2023-12-19T18:40:00.203 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.646+0000 7fa750b68700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa738039fb0 0x7fa73803c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.647+0000 7fa750b68700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa738039fb0 0x7fa73803c390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa7400039d0 tx=0x7fa7400092f0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.648+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 <== mgr.14160 v2:172.21.15.59:6800/6145510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fa738039200 con 0x7fa738039fb0 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.654+0000 7fa7535cd700 1 -- 172.21.15.59:0/3827912418 --> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa74c04bb80 con 0x7fa738039fb0 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.654+0000 7fa73dffb700 1 -- 172.21.15.59:0/3827912418 <== mgr.14160 v2:172.21.15.59:6800/6145510 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fa74c04bb80 con 0x7fa738039fb0 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 -- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa738039fb0 msgr2=0x7fa73803c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa738039fb0 0x7fa73803c390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa7400039d0 tx=0x7fa7400092f0 comp rx=0 tx=0).stop 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 -- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 msgr2=0x7fa74c113ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c113ea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa748011b60 tx=0x7fa748009250 comp rx=0 tx=0).stop 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 -- 172.21.15.59:0/3827912418 shutdown_connections 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa738039fb0 0x7fa73803c390 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa74c1027b0 0x7fa74c113ea0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 --2- 172.21.15.59:0/3827912418 >> [v2:172.21.15.59:6800/2067989811,v1:172.21.15.59:6801/2067989811] conn(0x7fa738036640 0x7fa738038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 -- 172.21.15.59:0/3827912418 >> 172.21.15.59:0/3827912418 conn(0x7fa74c0fc0b0 msgr2=0x7fa74c0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:00.204 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 -- 172.21.15.59:0/3827912418 shutdown_connections 2023-12-19T18:40:00.205 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:39:59.655+0000 7fa7535cd700 1 -- 172.21.15.59:0/3827912418 wait complete. 2023-12-19T18:40:00.205 INFO:teuthology.orchestra.run.smithi059.stdout:mgr epoch 9 is available 2023-12-19T18:40:00.205 INFO:teuthology.orchestra.run.smithi059.stdout:Generating a dashboard self-signed certificate... 2023-12-19T18:40:00.868 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:00 smithi059 ceph-mon[80661]: [19/Dec/2023:18:39:59] ENGINE Bus STARTING 2023-12-19T18:40:00.868 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:00 smithi059 ceph-mon[80661]: [19/Dec/2023:18:39:59] ENGINE Serving on https://172.21.15.59:7150 2023-12-19T18:40:00.868 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:00 smithi059 ceph-mon[80661]: [19/Dec/2023:18:39:59] ENGINE Bus STARTED 2023-12-19T18:40:00.868 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:00 smithi059 ceph-mon[80661]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-19T18:40:00.868 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:00 smithi059 ceph-mon[80661]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-19T18:40:00.869 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:00 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:01.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-12-19T18:40:01.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.613+0000 7f7277aec700 1 Processor -- start 2023-12-19T18:40:01.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.613+0000 7f7277aec700 1 -- start start 2023-12-19T18:40:01.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.613+0000 7f7277aec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f7270104f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:01.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.613+0000 7f7277aec700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7270105460 con 0x7f7270076be0 2023-12-19T18:40:01.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.613+0000 7f7275888700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f7270104f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:01.539 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.613+0000 7f7275888700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f7270104f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45322/0 (socket says 172.21.15.59:45322) 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.613+0000 7f7275888700 1 -- 172.21.15.59:0/2847494989 learned_addr learned my addr 172.21.15.59:0/2847494989 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.614+0000 7f7275888700 1 -- 172.21.15.59:0/2847494989 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72701055a0 con 0x7f7270076be0 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.614+0000 7f7275888700 1 --2- 172.21.15.59:0/2847494989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f7270104f20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f726c0098e0 tx=0x7f726c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae18965ae8146b6 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.614+0000 7f7274886700 1 -- 172.21.15.59:0/2847494989 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f726c00b860 con 0x7f7270076be0 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.614+0000 7f7274886700 1 -- 172.21.15.59:0/2847494989 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f726c00be70 con 0x7f7270076be0 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.614+0000 7f7274886700 1 -- 172.21.15.59:0/2847494989 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f726c010380 con 0x7f7270076be0 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.614+0000 7f7277aec700 1 -- 172.21.15.59:0/2847494989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 msgr2=0x7f7270104f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.614+0000 7f7277aec700 1 --2- 172.21.15.59:0/2847494989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f7270104f20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f726c0098e0 tx=0x7f726c009250 comp rx=0 tx=0).stop 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7277aec700 1 -- 172.21.15.59:0/2847494989 shutdown_connections 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7277aec700 1 --2- 172.21.15.59:0/2847494989 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f7270104f20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7277aec700 1 -- 172.21.15.59:0/2847494989 >> 172.21.15.59:0/2847494989 conn(0x7f72700fc950 msgr2=0x7f72700fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:01.540 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7277aec700 1 -- 172.21.15.59:0/2847494989 shutdown_connections 2023-12-19T18:40:01.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7277aec700 1 -- 172.21.15.59:0/2847494989 wait complete. 2023-12-19T18:40:01.541 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7277aec700 1 Processor -- start 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7277aec700 1 -- start start 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7277aec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f72701029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7277aec700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7270100e20 con 0x7f7270076be0 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.615+0000 7f7275888700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f72701029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.616+0000 7f7275888700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f72701029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45336/0 (socket says 172.21.15.59:45336) 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.616+0000 7f7275888700 1 -- 172.21.15.59:0/120332906 learned_addr learned my addr 172.21.15.59:0/120332906 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.616+0000 7f7275888700 1 -- 172.21.15.59:0/120332906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c0094f0 con 0x7f7270076be0 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.616+0000 7f7275888700 1 --2- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f72701029a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f726c006d40 tx=0x7f726c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.616+0000 7f72667fc700 1 -- 172.21.15.59:0/120332906 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f726c010960 con 0x7f7270076be0 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.616+0000 7f72667fc700 1 -- 172.21.15.59:0/120332906 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f726c01a430 con 0x7f7270076be0 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.616+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7270101020 con 0x7f7270076be0 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.616+0000 7f72667fc700 1 -- 172.21.15.59:0/120332906 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f726c0245b0 con 0x7f7270076be0 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.616+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7270101430 con 0x7f7270076be0 2023-12-19T18:40:01.542 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.617+0000 7f72667fc700 1 -- 172.21.15.59:0/120332906 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7f726c02d3f0 con 0x7f7270076be0 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.617+0000 7f72667fc700 1 --2- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f725c036570 0x7f725c038a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.617+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727004bb80 con 0x7f7270076be0 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.617+0000 7f72667fc700 1 -- 172.21.15.59:0/120332906 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f726c04a190 con 0x7f7270076be0 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.617+0000 7f7275087700 1 --2- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f725c036570 0x7f725c038a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.618+0000 7f7275087700 1 --2- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f725c036570 0x7f725c038a20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7260009990 tx=0x7f7260006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.622+0000 7f72667fc700 1 -- 172.21.15.59:0/120332906 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f726c018ce0 con 0x7f7270076be0 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.710+0000 7f72667fc700 1 -- 172.21.15.59:0/120332906 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f726c01a9f0 con 0x7f7270076be0 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.868+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 --> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f727010adf0 con 0x7f725c036570 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.924+0000 7f72667fc700 1 -- 172.21.15.59:0/120332906 <== mgr.14160 v2:172.21.15.59:6800/6145510 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f727010adf0 con 0x7f725c036570 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f725c036570 msgr2=0x7f725c038a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 --2- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f725c036570 0x7f725c038a20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7260009990 tx=0x7f7260006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 msgr2=0x7f72701029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:01.543 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 --2- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f72701029a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f726c006d40 tx=0x7f726c009250 comp rx=0 tx=0).stop 2023-12-19T18:40:01.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 shutdown_connections 2023-12-19T18:40:01.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 --2- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f725c036570 0x7f725c038a20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:01.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 --2- 172.21.15.59:0/120332906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7270076be0 0x7f72701029a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:01.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 >> 172.21.15.59:0/120332906 conn(0x7f72700fc950 msgr2=0x7f72700fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:01.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 shutdown_connections 2023-12-19T18:40:01.544 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:00.926+0000 7f7277aec700 1 -- 172.21.15.59:0/120332906 wait complete. 2023-12-19T18:40:01.544 INFO:teuthology.orchestra.run.smithi059.stdout:Creating initial admin user... 2023-12-19T18:40:01.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:01 smithi059 ceph-mon[80661]: mgrmap e12: smithi059.ccbnob(active, since 2s) 2023-12-19T18:40:01.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:01 smithi059 ceph-mon[80661]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:40:01.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:01 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:01.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:01 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:01.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:01 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:03.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$O0kXU5qZ9opC/fZf.neXU.NplLCM7dtnX9PSAG/.2Rr94.VzyezHy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1703011202, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-19T18:40:03.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.020+0000 7f42f85ed700 1 Processor -- start 2023-12-19T18:40:03.493 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.020+0000 7f42f85ed700 1 -- start start 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.020+0000 7f42f85ed700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f01015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.020+0000 7f42f85ed700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0101b20 con 0x7f42f01011d0 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.021+0000 7f42f6389700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f01015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.021+0000 7f42f6389700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f01015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36768/0 (socket says 172.21.15.59:36768) 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.021+0000 7f42f6389700 1 -- 172.21.15.59:0/2672763982 learned_addr learned my addr 172.21.15.59:0/2672763982 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.021+0000 7f42f6389700 1 -- 172.21.15.59:0/2672763982 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f0101c60 con 0x7f42f01011d0 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.021+0000 7f42f6389700 1 --2- 172.21.15.59:0/2672763982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f01015e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0098e0 tx=0x7f42ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ca21190760b58c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.021+0000 7f42f5387700 1 -- 172.21.15.59:0/2672763982 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42ec00b860 con 0x7f42f01011d0 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.021+0000 7f42f5387700 1 -- 172.21.15.59:0/2672763982 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42ec00be70 con 0x7f42f01011d0 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.021+0000 7f42f5387700 1 -- 172.21.15.59:0/2672763982 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42ec010380 con 0x7f42f01011d0 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.022+0000 7f42f85ed700 1 -- 172.21.15.59:0/2672763982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 msgr2=0x7f42f01015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.022+0000 7f42f85ed700 1 --2- 172.21.15.59:0/2672763982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f01015e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0098e0 tx=0x7f42ec009250 comp rx=0 tx=0).stop 2023-12-19T18:40:03.494 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.022+0000 7f42f85ed700 1 -- 172.21.15.59:0/2672763982 shutdown_connections 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.022+0000 7f42f85ed700 1 --2- 172.21.15.59:0/2672763982 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f01015e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.022+0000 7f42f85ed700 1 -- 172.21.15.59:0/2672763982 >> 172.21.15.59:0/2672763982 conn(0x7f42f00fc950 msgr2=0x7f42f00fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.022+0000 7f42f85ed700 1 -- 172.21.15.59:0/2672763982 shutdown_connections 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.022+0000 7f42f85ed700 1 -- 172.21.15.59:0/2672763982 wait complete. 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.023+0000 7f42f85ed700 1 Processor -- start 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.023+0000 7f42f85ed700 1 -- start start 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.023+0000 7f42f85ed700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f0116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.023+0000 7f42f85ed700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f0111c80 con 0x7f42f01011d0 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.023+0000 7f42f6389700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f0116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.023+0000 7f42f6389700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f0116290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36782/0 (socket says 172.21.15.59:36782) 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.023+0000 7f42f6389700 1 -- 172.21.15.59:0/1888398999 learned_addr learned my addr 172.21.15.59:0/1888398999 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.024+0000 7f42f6389700 1 -- 172.21.15.59:0/1888398999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec0094f0 con 0x7f42f01011d0 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.024+0000 7f42f6389700 1 --2- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f0116290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c010 tx=0x7f42ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.024+0000 7f42e6ffd700 1 -- 172.21.15.59:0/1888398999 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42ec010960 con 0x7f42f01011d0 2023-12-19T18:40:03.495 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.024+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f0076bf0 con 0x7f42f01011d0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.024+0000 7f42e6ffd700 1 -- 172.21.15.59:0/1888398999 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42ec01a430 con 0x7f42f01011d0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.024+0000 7f42e6ffd700 1 -- 172.21.15.59:0/1888398999 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42ec019580 con 0x7f42f01011d0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.024+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f01167d0 con 0x7f42f01011d0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.025+0000 7f42e6ffd700 1 -- 172.21.15.59:0/1888398999 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f42ec0197b0 con 0x7f42f01011d0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.025+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f010a090 con 0x7f42f01011d0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.025+0000 7f42e6ffd700 1 --2- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f42dc036690 0x7f42dc038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.026+0000 7f42e6ffd700 1 -- 172.21.15.59:0/1888398999 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f42ec04a490 con 0x7f42f01011d0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.026+0000 7f42f5b88700 1 --2- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f42dc036690 0x7f42dc038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.026+0000 7f42f5b88700 1 --2- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f42dc036690 0x7f42dc038b40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009990 tx=0x7f42e0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.034+0000 7f42e6ffd700 1 -- 172.21.15.59:0/1888398999 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f42ec02b950 con 0x7f42f01011d0 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.253+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 --> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f42f005f090 con 0x7f42dc036690 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.500+0000 7f42e6ffd700 1 -- 172.21.15.59:0/1888398999 <== mgr.14160 v2:172.21.15.59:6800/6145510 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f42f005f090 con 0x7f42dc036690 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f42dc036690 msgr2=0x7f42dc038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:03.496 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 --2- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f42dc036690 0x7f42dc038b40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f42e0009990 tx=0x7f42e0006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:03.497 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 msgr2=0x7f42f0116290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:03.497 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 --2- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f0116290 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00c010 tx=0x7f42ec009250 comp rx=0 tx=0).stop 2023-12-19T18:40:03.497 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 shutdown_connections 2023-12-19T18:40:03.497 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 --2- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f42dc036690 0x7f42dc038b40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:03.497 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 --2- 172.21.15.59:0/1888398999 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42f01011d0 0x7f42f0116290 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:03.497 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 >> 172.21.15.59:0/1888398999 conn(0x7f42f00fc950 msgr2=0x7f42f00fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:03.497 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 shutdown_connections 2023-12-19T18:40:03.497 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:02.502+0000 7f42f85ed700 1 -- 172.21.15.59:0/1888398999 wait complete. 2023-12-19T18:40:03.497 INFO:teuthology.orchestra.run.smithi059.stdout:Fetching dashboard port number... 2023-12-19T18:40:03.805 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:03 smithi059 ceph-mon[80661]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:40:03.805 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:03 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:04.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stdout 8443 2023-12-19T18:40:04.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.911+0000 7f6c98519700 1 Processor -- start 2023-12-19T18:40:04.658 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.912+0000 7f6c98519700 1 -- start start 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.912+0000 7f6c98519700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c90102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.912+0000 7f6c98519700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90103100 con 0x7f6c901027b0 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.912+0000 7f6c962b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c90102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.912+0000 7f6c962b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c90102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36794/0 (socket says 172.21.15.59:36794) 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.912+0000 7f6c962b5700 1 -- 172.21.15.59:0/3485519388 learned_addr learned my addr 172.21.15.59:0/3485519388 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.912+0000 7f6c962b5700 1 -- 172.21.15.59:0/3485519388 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c90103240 con 0x7f6c901027b0 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c962b5700 1 --2- 172.21.15.59:0/3485519388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c90102bc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0098e0 tx=0x7f6c8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3461d53db46716f4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c952b3700 1 -- 172.21.15.59:0/3485519388 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c8c00b860 con 0x7f6c901027b0 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c952b3700 1 -- 172.21.15.59:0/3485519388 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c8c00be70 con 0x7f6c901027b0 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c952b3700 1 -- 172.21.15.59:0/3485519388 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c8c010380 con 0x7f6c901027b0 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c98519700 1 -- 172.21.15.59:0/3485519388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 msgr2=0x7f6c90102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c98519700 1 --2- 172.21.15.59:0/3485519388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c90102bc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0098e0 tx=0x7f6c8c009250 comp rx=0 tx=0).stop 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c98519700 1 -- 172.21.15.59:0/3485519388 shutdown_connections 2023-12-19T18:40:04.659 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c98519700 1 --2- 172.21.15.59:0/3485519388 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c90102bc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c98519700 1 -- 172.21.15.59:0/3485519388 >> 172.21.15.59:0/3485519388 conn(0x7f6c900fc0b0 msgr2=0x7f6c900fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c98519700 1 -- 172.21.15.59:0/3485519388 shutdown_connections 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.913+0000 7f6c98519700 1 -- 172.21.15.59:0/3485519388 wait complete. 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.914+0000 7f6c98519700 1 Processor -- start 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.914+0000 7f6c98519700 1 -- start start 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.914+0000 7f6c98519700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c901133b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.914+0000 7f6c98519700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901138f0 con 0x7f6c901027b0 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.914+0000 7f6c962b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c901133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.914+0000 7f6c962b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c901133b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36796/0 (socket says 172.21.15.59:36796) 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.914+0000 7f6c962b5700 1 -- 172.21.15.59:0/2355982578 learned_addr learned my addr 172.21.15.59:0/2355982578 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.915+0000 7f6c962b5700 1 -- 172.21.15.59:0/2355982578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8c0094f0 con 0x7f6c901027b0 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.915+0000 7f6c962b5700 1 --2- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c901133b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c011b60 tx=0x7f6c8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.915+0000 7f6c82ffd700 1 -- 172.21.15.59:0/2355982578 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c8c010850 con 0x7f6c901027b0 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.915+0000 7f6c82ffd700 1 -- 172.21.15.59:0/2355982578 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c8c010e60 con 0x7f6c901027b0 2023-12-19T18:40:04.660 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.915+0000 7f6c82ffd700 1 -- 172.21.15.59:0/2355982578 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c8c01ae20 con 0x7f6c901027b0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.915+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c90113af0 con 0x7f6c901027b0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.915+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c90113f00 con 0x7f6c901027b0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.916+0000 7f6c82ffd700 1 -- 172.21.15.59:0/2355982578 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f6c8c0109b0 con 0x7f6c901027b0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.916+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9004bb80 con 0x7f6c901027b0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.916+0000 7f6c82ffd700 1 --2- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f6c7c036620 0x7f6c7c038ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.916+0000 7f6c82ffd700 1 -- 172.21.15.59:0/2355982578 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6c8c04a360 con 0x7f6c901027b0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.916+0000 7f6c95ab4700 1 --2- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f6c7c036620 0x7f6c7c038ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.917+0000 7f6c95ab4700 1 --2- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f6c7c036620 0x7f6c7c038ad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009990 tx=0x7f6c84006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:03.921+0000 7f6c82ffd700 1 -- 172.21.15.59:0/2355982578 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6c8c02b940 con 0x7f6c901027b0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.117+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f6c90101fe0 con 0x7f6c901027b0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.118+0000 7f6c82ffd700 1 -- 172.21.15.59:0/2355982578 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f6c8c02b760 con 0x7f6c901027b0 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f6c7c036620 msgr2=0x7f6c7c038ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:04.661 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 --2- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f6c7c036620 0x7f6c7c038ad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009990 tx=0x7f6c84006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:04.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 msgr2=0x7f6c901133b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:04.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 --2- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c901133b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c011b60 tx=0x7f6c8c009250 comp rx=0 tx=0).stop 2023-12-19T18:40:04.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 shutdown_connections 2023-12-19T18:40:04.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 --2- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f6c7c036620 0x7f6c7c038ad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:04.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 --2- 172.21.15.59:0/2355982578 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6c901027b0 0x7f6c901133b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:04.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 >> 172.21.15.59:0/2355982578 conn(0x7f6c900fc0b0 msgr2=0x7f6c90105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:04.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 shutdown_connections 2023-12-19T18:40:04.662 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:04.120+0000 7f6c98519700 1 -- 172.21.15.59:0/2355982578 wait complete. 2023-12-19T18:40:04.666 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-19T18:40:04.666 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout disabled 2023-12-19T18:40:04.674 INFO:teuthology.orchestra.run.smithi059.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-19T18:40:04.674 INFO:teuthology.orchestra.run.smithi059.stdout:systemctl: stdout inactive 2023-12-19T18:40:04.674 INFO:teuthology.orchestra.run.smithi059.stdout:firewalld.service is not enabled 2023-12-19T18:40:04.674 INFO:teuthology.orchestra.run.smithi059.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-19T18:40:04.676 INFO:teuthology.orchestra.run.smithi059.stdout:Ceph Dashboard is now available at: 2023-12-19T18:40:04.676 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:04.677 INFO:teuthology.orchestra.run.smithi059.stdout: URL: https://smithi059.front.sepia.ceph.com:8443/ 2023-12-19T18:40:04.677 INFO:teuthology.orchestra.run.smithi059.stdout: User: admin 2023-12-19T18:40:04.677 INFO:teuthology.orchestra.run.smithi059.stdout: Password: 08u9jbbh6y 2023-12-19T18:40:04.677 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:04.677 INFO:teuthology.orchestra.run.smithi059.stdout:Saving cluster configuration to /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config directory 2023-12-19T18:40:04.677 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling autotune for osd_memory_target 2023-12-19T18:40:04.802 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:04 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2355982578' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-19T18:40:05.839 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.084+0000 7fa851741700 1 Processor -- start 2023-12-19T18:40:05.839 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.084+0000 7fa851741700 1 -- start start 2023-12-19T18:40:05.839 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.084+0000 7fa851741700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:05.839 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.084+0000 7fa851741700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c101b20 con 0x7fa84c1011d0 2023-12-19T18:40:05.839 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.084+0000 7fa84affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:05.839 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.084+0000 7fa84affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36804/0 (socket says 172.21.15.59:36804) 2023-12-19T18:40:05.839 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.084+0000 7fa84affd700 1 -- 172.21.15.59:0/3628838039 learned_addr learned my addr 172.21.15.59:0/3628838039 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:05.839 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.085+0000 7fa84affd700 1 -- 172.21.15.59:0/3628838039 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c101c60 con 0x7fa84c1011d0 2023-12-19T18:40:05.839 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.085+0000 7fa84affd700 1 --2- 172.21.15.59:0/3628838039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1015e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009e50 tx=0x7fa83c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b185084a643125 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.085+0000 7fa849ffb700 1 -- 172.21.15.59:0/3628838039 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa83c0092e0 con 0x7fa84c1011d0 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.085+0000 7fa849ffb700 1 -- 172.21.15.59:0/3628838039 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa83c012430 con 0x7fa84c1011d0 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.085+0000 7fa849ffb700 1 -- 172.21.15.59:0/3628838039 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa83c0075c0 con 0x7fa84c1011d0 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.086+0000 7fa851741700 1 -- 172.21.15.59:0/3628838039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 msgr2=0x7fa84c1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.086+0000 7fa851741700 1 --2- 172.21.15.59:0/3628838039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1015e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009e50 tx=0x7fa83c009840 comp rx=0 tx=0).stop 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.086+0000 7fa851741700 1 -- 172.21.15.59:0/3628838039 shutdown_connections 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.086+0000 7fa851741700 1 --2- 172.21.15.59:0/3628838039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1015e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.086+0000 7fa851741700 1 -- 172.21.15.59:0/3628838039 >> 172.21.15.59:0/3628838039 conn(0x7fa84c0fc970 msgr2=0x7fa84c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.086+0000 7fa851741700 1 -- 172.21.15.59:0/3628838039 shutdown_connections 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.086+0000 7fa851741700 1 -- 172.21.15.59:0/3628838039 wait complete. 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa851741700 1 Processor -- start 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa851741700 1 -- start start 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa851741700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1111f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:05.840 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa851741700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c111730 con 0x7fa84c1011d0 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa84affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa84affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1111f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36816/0 (socket says 172.21.15.59:36816) 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa84affd700 1 -- 172.21.15.59:0/1304263414 learned_addr learned my addr 172.21.15.59:0/1304263414 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa84affd700 1 -- 172.21.15.59:0/1304263414 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c009ae0 con 0x7fa84c1011d0 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa84affd700 1 --2- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1111f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0127d0 tx=0x7fa83c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa8437fe700 1 -- 172.21.15.59:0/1304263414 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa83c0092e0 con 0x7fa84c1011d0 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.087+0000 7fa8437fe700 1 -- 172.21.15.59:0/1304263414 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa83c007ce0 con 0x7fa84c1011d0 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.088+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c111930 con 0x7fa84c1011d0 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.088+0000 7fa8437fe700 1 -- 172.21.15.59:0/1304263414 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa83c01c910 con 0x7fa84c1011d0 2023-12-19T18:40:05.841 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.088+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c1125d0 con 0x7fa84c1011d0 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.088+0000 7fa8437fe700 1 -- 172.21.15.59:0/1304263414 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fa83c019070 con 0x7fa84c1011d0 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.089+0000 7fa8437fe700 1 --2- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa838036640 0x7fa838038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.089+0000 7fa8437fe700 1 -- 172.21.15.59:0/1304263414 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa83c049f10 con 0x7fa84c1011d0 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.089+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c05f270 con 0x7fa84c1011d0 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.089+0000 7fa84a7fc700 1 --2- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa838036640 0x7fa838038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.089+0000 7fa84a7fc700 1 --2- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa838036640 0x7fa838038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa834009990 tx=0x7fa834006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.093+0000 7fa8437fe700 1 -- 172.21.15.59:0/1304263414 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa83c017020 con 0x7fa84c1011d0 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.290+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fa84c04bb80 con 0x7fa84c1011d0 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.291+0000 7fa8437fe700 1 -- 172.21.15.59:0/1304263414 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fa83c019910 con 0x7fa84c1011d0 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa838036640 msgr2=0x7fa838038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 --2- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa838036640 0x7fa838038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa834009990 tx=0x7fa834006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:05.842 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 msgr2=0x7fa84c1111f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:05.843 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 --2- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1111f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0127d0 tx=0x7fa83c009840 comp rx=0 tx=0).stop 2023-12-19T18:40:05.843 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 shutdown_connections 2023-12-19T18:40:05.843 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 --2- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fa838036640 0x7fa838038af0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:05.843 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 --2- 172.21.15.59:0/1304263414 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa84c1011d0 0x7fa84c1111f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:05.843 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 >> 172.21.15.59:0/1304263414 conn(0x7fa84c0fc970 msgr2=0x7fa84c0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:05.843 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 shutdown_connections 2023-12-19T18:40:05.843 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:05.293+0000 7fa851741700 1 -- 172.21.15.59:0/1304263414 wait complete. 2023-12-19T18:40:06.862 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:06 smithi059 ceph-mon[80661]: mgrmap e13: smithi059.ccbnob(active, since 6s) 2023-12-19T18:40:07.113 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.244+0000 7f03ec555700 1 Processor -- start 2023-12-19T18:40:07.113 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.244+0000 7f03ec555700 1 -- start start 2023-12-19T18:40:07.113 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.245+0000 7f03ec555700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:07.113 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.245+0000 7f03ec555700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e4077770 con 0x7f03e4078e20 2023-12-19T18:40:07.113 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.245+0000 7f03ea2f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:07.113 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.245+0000 7f03ea2f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36822/0 (socket says 172.21.15.59:36822) 2023-12-19T18:40:07.113 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.245+0000 7f03ea2f1700 1 -- 172.21.15.59:0/746026473 learned_addr learned my addr 172.21.15.59:0/746026473 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:07.114 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.245+0000 7f03ea2f1700 1 -- 172.21.15.59:0/746026473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e40778b0 con 0x7f03e4078e20 2023-12-19T18:40:07.114 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.245+0000 7f03ea2f1700 1 --2- 172.21.15.59:0/746026473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4077230 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f03d40098e0 tx=0x7f03d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3c888fe16618a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:07.114 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03e92ef700 1 -- 172.21.15.59:0/746026473 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d400b860 con 0x7f03e4078e20 2023-12-19T18:40:07.114 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03e92ef700 1 -- 172.21.15.59:0/746026473 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03d400be70 con 0x7f03e4078e20 2023-12-19T18:40:07.114 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03e92ef700 1 -- 172.21.15.59:0/746026473 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d4010380 con 0x7f03e4078e20 2023-12-19T18:40:07.114 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03ec555700 1 -- 172.21.15.59:0/746026473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 msgr2=0x7f03e4077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:07.114 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03ec555700 1 --2- 172.21.15.59:0/746026473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4077230 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f03d40098e0 tx=0x7f03d4009250 comp rx=0 tx=0).stop 2023-12-19T18:40:07.114 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03ec555700 1 -- 172.21.15.59:0/746026473 shutdown_connections 2023-12-19T18:40:07.115 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03ec555700 1 --2- 172.21.15.59:0/746026473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4077230 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:07.115 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03ec555700 1 -- 172.21.15.59:0/746026473 >> 172.21.15.59:0/746026473 conn(0x7f03e40fda40 msgr2=0x7f03e40ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:07.115 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03ec555700 1 -- 172.21.15.59:0/746026473 shutdown_connections 2023-12-19T18:40:07.115 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.246+0000 7f03ec555700 1 -- 172.21.15.59:0/746026473 wait complete. 2023-12-19T18:40:07.115 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.247+0000 7f03ec555700 1 Processor -- start 2023-12-19T18:40:07.115 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.247+0000 7f03ec555700 1 -- start start 2023-12-19T18:40:07.115 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.247+0000 7f03ec555700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4113360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:07.116 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.247+0000 7f03ec555700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e41138a0 con 0x7f03e4078e20 2023-12-19T18:40:07.116 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.247+0000 7f03ea2f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4113360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:07.116 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.247+0000 7f03ea2f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4113360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36824/0 (socket says 172.21.15.59:36824) 2023-12-19T18:40:07.116 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.247+0000 7f03ea2f1700 1 -- 172.21.15.59:0/2128198773 learned_addr learned my addr 172.21.15.59:0/2128198773 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:07.116 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.247+0000 7f03ea2f1700 1 -- 172.21.15.59:0/2128198773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d40094f0 con 0x7f03e4078e20 2023-12-19T18:40:07.116 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.248+0000 7f03ea2f1700 1 --2- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4113360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f03d400ba70 tx=0x7f03d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:07.116 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.248+0000 7f03daffd700 1 -- 172.21.15.59:0/2128198773 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d4010960 con 0x7f03e4078e20 2023-12-19T18:40:07.116 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.248+0000 7f03daffd700 1 -- 172.21.15.59:0/2128198773 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03d4024430 con 0x7f03e4078e20 2023-12-19T18:40:07.116 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.248+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e4113aa0 con 0x7f03e4078e20 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.248+0000 7f03daffd700 1 -- 172.21.15.59:0/2128198773 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d401a5f0 con 0x7f03e4078e20 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.248+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e4113eb0 con 0x7f03e4078e20 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.249+0000 7f03daffd700 1 -- 172.21.15.59:0/2128198773 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f03d401a860 con 0x7f03e4078e20 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.249+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e4108620 con 0x7f03e4078e20 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.249+0000 7f03daffd700 1 --2- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f03d0036730 0x7f03d0038be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.249+0000 7f03daffd700 1 -- 172.21.15.59:0/2128198773 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f03d404a640 con 0x7f03e4078e20 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.249+0000 7f03e9af0700 1 --2- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f03d0036730 0x7f03d0038be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.249+0000 7f03e9af0700 1 --2- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f03d0036730 0x7f03d0038be0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009990 tx=0x7f03e0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.254+0000 7f03daffd700 1 -- 172.21.15.59:0/2128198773 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f03d4018d90 con 0x7f03e4078e20 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.559+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f03e410cfb0 con 0x7f03e4078e20 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.564+0000 7f03daffd700 1 -- 172.21.15.59:0/2128198773 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f03d4018350 con 0x7f03e4078e20 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f03d0036730 msgr2=0x7f03d0038be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:07.117 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 --2- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f03d0036730 0x7f03d0038be0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03e0009990 tx=0x7f03e0006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 msgr2=0x7f03e4113360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 --2- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4113360 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f03d400ba70 tx=0x7f03d4009250 comp rx=0 tx=0).stop 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 shutdown_connections 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 --2- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f03d0036730 0x7f03d0038be0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 --2- 172.21.15.59:0/2128198773 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03e4078e20 0x7f03e4113360 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 >> 172.21.15.59:0/2128198773 conn(0x7f03e40fda40 msgr2=0x7f03e40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 shutdown_connections 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr 2023-12-19T18:40:06.566+0000 7f03ec555700 1 -- 172.21.15.59:0/2128198773 wait complete. 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout:Or, if you are only running a single cluster on this host: 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:07.118 INFO:teuthology.orchestra.run.smithi059.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout:Please consider enabling telemetry to help improve Ceph: 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout: ceph telemetry on 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout:For more information see: 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:07.119 INFO:teuthology.orchestra.run.smithi059.stdout:Bootstrap complete. 2023-12-19T18:40:07.155 INFO:tasks.cephadm:Fetching config... 2023-12-19T18:40:07.156 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:40:07.156 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-19T18:40:07.172 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-19T18:40:07.172 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:40:07.172 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-19T18:40:07.228 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-19T18:40:07.228 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:40:07.228 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/keyring of=/dev/stdout 2023-12-19T18:40:07.299 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-19T18:40:07.299 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:40:07.299 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-19T18:40:07.355 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-19T18:40:07.356 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcZ741Eu8uRFJIzSVCG5g7wC+O6CkKG23f2dBASTDklVCRhViodR92fqpcBApSuyqD81p6nRoUvVgu29pS5q4J2KCWq2plNgEBnbBwinjbdjdmpFEPde+ze6U4vl2KcTs7kkK6SqHoZHwd8JkNzM4x+Fr+ijaNFVeyQzoNALsX2K5IYvpS7/PHkmN7nmhjNWpzsbyqDa0hWiC8qc0fWon4j1eS0pNV3RRuSrUdMAAAgPZMEZelXAmZRkgaXKjzCglomn2Ig1BcRiydUM+1iUz9Mhu6EAlvtFBDKT4VtgmQCs4Jn2tfOl2rti482QZaCp9iNhsO1q6G36O+nTN6vebHW5l6IcmegfozshZa8/s4Zx61r+XWmKGFOZjdOHxt4qp3VXMnlN/8aPTmd1oPpQHFID4p7zGlysKcj85L3iGoL0WIRvf+g1uOVSddpUJvAx+zvd0dSyRKZLyw+4ZiJvHImwyVeh8DSELGOeN/uWzFilrQ4lHBl1Lm/IYGtHPswhM= ceph-bf854274-9e9d-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-19T18:40:07.441 INFO:teuthology.orchestra.run.smithi059.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcZ741Eu8uRFJIzSVCG5g7wC+O6CkKG23f2dBASTDklVCRhViodR92fqpcBApSuyqD81p6nRoUvVgu29pS5q4J2KCWq2plNgEBnbBwinjbdjdmpFEPde+ze6U4vl2KcTs7kkK6SqHoZHwd8JkNzM4x+Fr+ijaNFVeyQzoNALsX2K5IYvpS7/PHkmN7nmhjNWpzsbyqDa0hWiC8qc0fWon4j1eS0pNV3RRuSrUdMAAAgPZMEZelXAmZRkgaXKjzCglomn2Ig1BcRiydUM+1iUz9Mhu6EAlvtFBDKT4VtgmQCs4Jn2tfOl2rti482QZaCp9iNhsO1q6G36O+nTN6vebHW5l6IcmegfozshZa8/s4Zx61r+XWmKGFOZjdOHxt4qp3VXMnlN/8aPTmd1oPpQHFID4p7zGlysKcj85L3iGoL0WIRvf+g1uOVSddpUJvAx+zvd0dSyRKZLyw+4ZiJvHImwyVeh8DSELGOeN/uWzFilrQ4lHBl1Lm/IYGtHPswhM= ceph-bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:40:07.460 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcZ741Eu8uRFJIzSVCG5g7wC+O6CkKG23f2dBASTDklVCRhViodR92fqpcBApSuyqD81p6nRoUvVgu29pS5q4J2KCWq2plNgEBnbBwinjbdjdmpFEPde+ze6U4vl2KcTs7kkK6SqHoZHwd8JkNzM4x+Fr+ijaNFVeyQzoNALsX2K5IYvpS7/PHkmN7nmhjNWpzsbyqDa0hWiC8qc0fWon4j1eS0pNV3RRuSrUdMAAAgPZMEZelXAmZRkgaXKjzCglomn2Ig1BcRiydUM+1iUz9Mhu6EAlvtFBDKT4VtgmQCs4Jn2tfOl2rti482QZaCp9iNhsO1q6G36O+nTN6vebHW5l6IcmegfozshZa8/s4Zx61r+XWmKGFOZjdOHxt4qp3VXMnlN/8aPTmd1oPpQHFID4p7zGlysKcj85L3iGoL0WIRvf+g1uOVSddpUJvAx+zvd0dSyRKZLyw+4ZiJvHImwyVeh8DSELGOeN/uWzFilrQ4lHBl1Lm/IYGtHPswhM= ceph-bf854274-9e9d-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-19T18:40:07.507 INFO:teuthology.orchestra.run.smithi089.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcZ741Eu8uRFJIzSVCG5g7wC+O6CkKG23f2dBASTDklVCRhViodR92fqpcBApSuyqD81p6nRoUvVgu29pS5q4J2KCWq2plNgEBnbBwinjbdjdmpFEPde+ze6U4vl2KcTs7kkK6SqHoZHwd8JkNzM4x+Fr+ijaNFVeyQzoNALsX2K5IYvpS7/PHkmN7nmhjNWpzsbyqDa0hWiC8qc0fWon4j1eS0pNV3RRuSrUdMAAAgPZMEZelXAmZRkgaXKjzCglomn2Ig1BcRiydUM+1iUz9Mhu6EAlvtFBDKT4VtgmQCs4Jn2tfOl2rti482QZaCp9iNhsO1q6G36O+nTN6vebHW5l6IcmegfozshZa8/s4Zx61r+XWmKGFOZjdOHxt4qp3VXMnlN/8aPTmd1oPpQHFID4p7zGlysKcj85L3iGoL0WIRvf+g1uOVSddpUJvAx+zvd0dSyRKZLyw+4ZiJvHImwyVeh8DSELGOeN/uWzFilrQ4lHBl1Lm/IYGtHPswhM= ceph-bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:40:07.523 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-19T18:40:07.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:07 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2128198773' entity='client.admin' 2023-12-19T18:40:07.788 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:40:10.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.154+0000 7fc561cd9700 1 -- 172.21.15.59:0/1896690529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 msgr2=0x7fc55c0fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:10.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.154+0000 7fc561cd9700 1 --2- 172.21.15.59:0/1896690529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 0x7fc55c0fcc80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc554009c60 tx=0x7fc554004d10 comp rx=0 tx=0).stop 2023-12-19T18:40:10.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.154+0000 7fc561cd9700 1 -- 172.21.15.59:0/1896690529 shutdown_connections 2023-12-19T18:40:10.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.154+0000 7fc561cd9700 1 --2- 172.21.15.59:0/1896690529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 0x7fc55c0fcc80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:10.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.154+0000 7fc561cd9700 1 -- 172.21.15.59:0/1896690529 >> 172.21.15.59:0/1896690529 conn(0x7fc55c0f7f50 msgr2=0x7fc55c0fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:10.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.154+0000 7fc561cd9700 1 -- 172.21.15.59:0/1896690529 shutdown_connections 2023-12-19T18:40:10.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.154+0000 7fc561cd9700 1 -- 172.21.15.59:0/1896690529 wait complete. 2023-12-19T18:40:10.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.154+0000 7fc561cd9700 1 Processor -- start 2023-12-19T18:40:10.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.155+0000 7fc561cd9700 1 -- start start 2023-12-19T18:40:10.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.155+0000 7fc561cd9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 0x7fc55c10cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:10.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.155+0000 7fc561cd9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c10d0e0 con 0x7fc55c0fc870 2023-12-19T18:40:10.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.155+0000 7fc55b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 0x7fc55c10cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.155+0000 7fc55b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 0x7fc55c10cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36846/0 (socket says 172.21.15.59:36846) 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.155+0000 7fc55b7fe700 1 -- 172.21.15.59:0/1628012442 learned_addr learned my addr 172.21.15.59:0/1628012442 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.155+0000 7fc55b7fe700 1 -- 172.21.15.59:0/1628012442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc554009940 con 0x7fc55c0fc870 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.155+0000 7fc55b7fe700 1 --2- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 0x7fc55c10cba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc554000f80 tx=0x7fc554004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.156+0000 7fc553fff700 1 -- 172.21.15.59:0/1628012442 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc55400bec0 con 0x7fc55c0fc870 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.156+0000 7fc553fff700 1 -- 172.21.15.59:0/1628012442 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc55401d8f0 con 0x7fc55c0fc870 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.156+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c10d2e0 con 0x7fc55c0fc870 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.156+0000 7fc553fff700 1 -- 172.21.15.59:0/1628012442 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc554012890 con 0x7fc55c0fc870 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.156+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c10df80 con 0x7fc55c0fc870 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.156+0000 7fc553fff700 1 -- 172.21.15.59:0/1628012442 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fc554019040 con 0x7fc55c0fc870 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.157+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c05f270 con 0x7fc55c0fc870 2023-12-19T18:40:10.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.157+0000 7fc553fff700 1 --2- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fc5480366d0 0x7fc548038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:10.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.157+0000 7fc553fff700 1 -- 172.21.15.59:0/1628012442 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc55404a1d0 con 0x7fc55c0fc870 2023-12-19T18:40:10.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.157+0000 7fc55affd700 1 --2- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fc5480366d0 0x7fc548038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:10.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.157+0000 7fc55affd700 1 --2- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fc5480366d0 0x7fc548038b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc544009990 tx=0x7fc544006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:10.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.162+0000 7fc553fff700 1 -- 172.21.15.59:0/1628012442 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc55402ebd0 con 0x7fc55c0fc870 2023-12-19T18:40:10.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:10 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:10.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:10 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:10.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:10 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:40:10.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:10 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:10.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:10 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T18:40:10.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:10 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-19T18:40:10.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:10 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:40:10.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:10 smithi059 ceph-mon[80661]: Deploying daemon crash.smithi059 on smithi059 2023-12-19T18:40:10.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.359+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc55c10eaf0 con 0x7fc55c0fc870 2023-12-19T18:40:10.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.363+0000 7fc553fff700 1 -- 172.21.15.59:0/1628012442 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc554021070 con 0x7fc55c0fc870 2023-12-19T18:40:10.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fc5480366d0 msgr2=0x7fc548038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:10.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 --2- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fc5480366d0 0x7fc548038b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc544009990 tx=0x7fc544006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:10.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 msgr2=0x7fc55c10cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:10.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 --2- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 0x7fc55c10cba0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc554000f80 tx=0x7fc554004d10 comp rx=0 tx=0).stop 2023-12-19T18:40:10.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 shutdown_connections 2023-12-19T18:40:10.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 --2- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fc5480366d0 0x7fc548038b80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:10.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 --2- 172.21.15.59:0/1628012442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc55c0fc870 0x7fc55c10cba0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:10.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 >> 172.21.15.59:0/1628012442 conn(0x7fc55c0f7f50 msgr2=0x7fc55c0fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:10.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 shutdown_connections 2023-12-19T18:40:10.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:10.365+0000 7fc561cd9700 1 -- 172.21.15.59:0/1628012442 wait complete. 2023-12-19T18:40:10.973 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-12-19T18:40:10.974 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-12-19T18:40:11.574 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:11 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1628012442' entity='client.admin' 2023-12-19T18:40:11.671 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:40:14.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.528+0000 7f0a6d31d700 1 -- 172.21.15.59:0/2286558842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 msgr2=0x7f0a68101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:14.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.528+0000 7f0a6d31d700 1 --2- 172.21.15.59:0/2286558842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 0x7f0a68101820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0a580048c0 tx=0x7f0a58009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:14.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.528+0000 7f0a6d31d700 1 -- 172.21.15.59:0/2286558842 shutdown_connections 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.528+0000 7f0a6d31d700 1 --2- 172.21.15.59:0/2286558842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 0x7f0a68101820 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.528+0000 7f0a6d31d700 1 -- 172.21.15.59:0/2286558842 >> 172.21.15.59:0/2286558842 conn(0x7f0a680f6e50 msgr2=0x7f0a680f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.528+0000 7f0a6d31d700 1 -- 172.21.15.59:0/2286558842 shutdown_connections 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.528+0000 7f0a6d31d700 1 -- 172.21.15.59:0/2286558842 wait complete. 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.529+0000 7f0a6d31d700 1 Processor -- start 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.529+0000 7f0a6d31d700 1 -- start start 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.529+0000 7f0a6d31d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 0x7f0a6810ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.529+0000 7f0a6d31d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6810f270 con 0x7f0a68066330 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.529+0000 7f0a66ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 0x7f0a6810ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.529+0000 7f0a66ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 0x7f0a6810ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36028/0 (socket says 172.21.15.59:36028) 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.529+0000 7f0a66ffd700 1 -- 172.21.15.59:0/1791875432 learned_addr learned my addr 172.21.15.59:0/1791875432 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.529+0000 7f0a66ffd700 1 -- 172.21.15.59:0/1791875432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a58005040 con 0x7f0a68066330 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.529+0000 7f0a66ffd700 1 --2- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 0x7f0a6810ed30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009040 tx=0x7f0a58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.530+0000 7f0a4f7fe700 1 -- 172.21.15.59:0/1791875432 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a58009620 con 0x7f0a68066330 2023-12-19T18:40:14.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.530+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6810f470 con 0x7f0a68066330 2023-12-19T18:40:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.530+0000 7f0a4f7fe700 1 -- 172.21.15.59:0/1791875432 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a58009780 con 0x7f0a68066330 2023-12-19T18:40:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.530+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6810ffd0 con 0x7f0a68066330 2023-12-19T18:40:14.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.530+0000 7f0a4f7fe700 1 -- 172.21.15.59:0/1791875432 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a5801c5b0 con 0x7f0a68066330 2023-12-19T18:40:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.531+0000 7f0a4f7fe700 1 -- 172.21.15.59:0/1791875432 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f0a58013a60 con 0x7f0a68066330 2023-12-19T18:40:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.531+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6804bb80 con 0x7f0a68066330 2023-12-19T18:40:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.531+0000 7f0a4f7fe700 1 --2- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0a50036720 0x7f0a50038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.531+0000 7f0a4f7fe700 1 -- 172.21.15.59:0/1791875432 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0a580497d0 con 0x7f0a68066330 2023-12-19T18:40:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.531+0000 7f0a667fc700 1 --2- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0a50036720 0x7f0a50038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:14.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.532+0000 7f0a667fc700 1 --2- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0a50036720 0x7f0a50038bd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009990 tx=0x7f0a5c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:14.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.536+0000 7f0a4f7fe700 1 -- 172.21.15.59:0/1791875432 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0a580119c0 con 0x7f0a68066330 2023-12-19T18:40:14.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.786+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 --> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a68110160 con 0x7f0a50036720 2023-12-19T18:40:14.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.789+0000 7f0a4f7fe700 1 -- 172.21.15.59:0/1791875432 <== mgr.14160 v2:172.21.15.59:6800/6145510 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0a68110160 con 0x7f0a50036720 2023-12-19T18:40:14.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.791+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0a50036720 msgr2=0x7f0a50038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:14.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.791+0000 7f0a6d31d700 1 --2- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0a50036720 0x7f0a50038bd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009990 tx=0x7f0a5c006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:14.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.791+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 msgr2=0x7f0a6810ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:14.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.791+0000 7f0a6d31d700 1 --2- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 0x7f0a6810ed30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009040 tx=0x7f0a58009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:14.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.791+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 shutdown_connections 2023-12-19T18:40:14.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.791+0000 7f0a6d31d700 1 --2- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0a50036720 0x7f0a50038bd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:14.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.791+0000 7f0a6d31d700 1 --2- 172.21.15.59:0/1791875432 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0a68066330 0x7f0a6810ed30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:14.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.791+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 >> 172.21.15.59:0/1791875432 conn(0x7f0a680f6e50 msgr2=0x7f0a680f7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:14.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.791+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 shutdown_connections 2023-12-19T18:40:14.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:14.792+0000 7f0a6d31d700 1 -- 172.21.15.59:0/1791875432 wait complete. 2023-12-19T18:40:15.469 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi089 2023-12-19T18:40:15.470 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:40:15.470 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/etc/ceph/ceph.conf 2023-12-19T18:40:15.489 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:40:15.489 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-19T18:40:15.546 INFO:tasks.cephadm:Adding host smithi089 to orchestrator... 2023-12-19T18:40:15.546 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch host add smithi089 2023-12-19T18:40:15.618 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:15 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:15.618 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:15 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:15.618 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:15 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:15.618 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:15 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:15.618 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:15 smithi059 ceph-mon[80661]: Deploying daemon node-exporter.smithi059 on smithi059 2023-12-19T18:40:15.618 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:15 smithi059 ceph-mon[80661]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:40:15.618 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:15 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:15.812 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:40:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.130+0000 7faa9e17f700 1 -- 172.21.15.59:0/2592821263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 msgr2=0x7faa980fee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.130+0000 7faa9e17f700 1 --2- 172.21.15.59:0/2592821263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 0x7faa980fee90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faa880048c0 tx=0x7faa88009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.131+0000 7faa9e17f700 1 -- 172.21.15.59:0/2592821263 shutdown_connections 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.131+0000 7faa9e17f700 1 --2- 172.21.15.59:0/2592821263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 0x7faa980fee90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.131+0000 7faa9e17f700 1 -- 172.21.15.59:0/2592821263 >> 172.21.15.59:0/2592821263 conn(0x7faa980fa250 msgr2=0x7faa980fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.131+0000 7faa9e17f700 1 -- 172.21.15.59:0/2592821263 shutdown_connections 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.131+0000 7faa9e17f700 1 -- 172.21.15.59:0/2592821263 wait complete. 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.131+0000 7faa9e17f700 1 Processor -- start 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.131+0000 7faa9e17f700 1 -- start start 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.131+0000 7faa9e17f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 0x7faa9810ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.131+0000 7faa9e17f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9810f280 con 0x7faa980fea80 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa977fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 0x7faa9810ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa977fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 0x7faa9810ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36044/0 (socket says 172.21.15.59:36044) 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa977fe700 1 -- 172.21.15.59:0/2004246765 learned_addr learned my addr 172.21.15.59:0/2004246765 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa977fe700 1 -- 172.21.15.59:0/2004246765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa88005040 con 0x7faa980fea80 2023-12-19T18:40:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa977fe700 1 --2- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 0x7faa9810ed40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa88009330 tx=0x7faa88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa7ffff700 1 -- 172.21.15.59:0/2004246765 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa880096b0 con 0x7faa980fea80 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9810f480 con 0x7faa980fea80 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa7ffff700 1 -- 172.21.15.59:0/2004246765 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa88009810 con 0x7faa980fea80 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa7ffff700 1 -- 172.21.15.59:0/2004246765 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8801c5b0 con 0x7faa980fea80 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.132+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa98110040 con 0x7faa980fea80 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.133+0000 7faa7ffff700 1 -- 172.21.15.59:0/2004246765 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7faa8801c870 con 0x7faa980fea80 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.133+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9805f270 con 0x7faa980fea80 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.133+0000 7faa7ffff700 1 --2- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7faa800366d0 0x7faa80038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.133+0000 7faa7ffff700 1 -- 172.21.15.59:0/2004246765 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7faa88049b90 con 0x7faa980fea80 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.133+0000 7faa96ffd700 1 --2- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7faa800366d0 0x7faa80038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.134+0000 7faa96ffd700 1 --2- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7faa800366d0 0x7faa80038b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009990 tx=0x7faa8c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.138+0000 7faa7ffff700 1 -- 172.21.15.59:0/2004246765 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7faa88011dd0 con 0x7faa980fea80 2023-12-19T18:40:18.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:18.339+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 --> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi089", "target": ["mon-mgr", ""]}) v1 -- 0x7faa98110cf0 con 0x7faa800366d0 2023-12-19T18:40:20.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:19 smithi059 ceph-mon[80661]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi089", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:40:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:19 smithi059 ceph-mon[80661]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:19 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:19 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:19 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:19 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:19 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:20.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.277+0000 7faa7ffff700 1 -- 172.21.15.59:0/2004246765 <== mgr.14160 v2:172.21.15.59:6800/6145510 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7faa98110cf0 con 0x7faa800366d0 2023-12-19T18:40:20.278 INFO:teuthology.orchestra.run.smithi059.stdout:Added host 'smithi089' with addr '172.21.15.89' 2023-12-19T18:40:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7faa800366d0 msgr2=0x7faa80038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 --2- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7faa800366d0 0x7faa80038b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009990 tx=0x7faa8c006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 msgr2=0x7faa9810ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 --2- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 0x7faa9810ed40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa88009330 tx=0x7faa88009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:20.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 shutdown_connections 2023-12-19T18:40:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 --2- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7faa800366d0 0x7faa80038b80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 --2- 172.21.15.59:0/2004246765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faa980fea80 0x7faa9810ed40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 >> 172.21.15.59:0/2004246765 conn(0x7faa980fa250 msgr2=0x7faa980fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 shutdown_connections 2023-12-19T18:40:20.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:20.279+0000 7faa9e17f700 1 -- 172.21.15.59:0/2004246765 wait complete. 2023-12-19T18:40:20.960 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:20 smithi059 ceph-mon[80661]: Deploying cephadm binary to smithi089 2023-12-19T18:40:20.960 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:20 smithi059 ceph-mon[80661]: Deploying daemon alertmanager.smithi059 on smithi059 2023-12-19T18:40:20.960 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:20 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:21.566 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch host ls --format=json 2023-12-19T18:40:21.853 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:40:21.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:21 smithi059 ceph-mon[80661]: Added host smithi089 2023-12-19T18:40:21.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:21 smithi059 ceph-mon[80661]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:24.116 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:23 smithi059 ceph-mon[80661]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:24.116 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:23 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 -- 172.21.15.59:0/2163296747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 msgr2=0x7fb5e80fd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 --2- 172.21.15.59:0/2163296747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 0x7fb5e80fd170 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40048c0 tx=0x7fb5e4009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 -- 172.21.15.59:0/2163296747 shutdown_connections 2023-12-19T18:40:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 --2- 172.21.15.59:0/2163296747 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 0x7fb5e80fd170 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 -- 172.21.15.59:0/2163296747 >> 172.21.15.59:0/2163296747 conn(0x7fb5e80f65d0 msgr2=0x7fb5e80f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 -- 172.21.15.59:0/2163296747 shutdown_connections 2023-12-19T18:40:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 -- 172.21.15.59:0/2163296747 wait complete. 2023-12-19T18:40:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 Processor -- start 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 -- start start 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 0x7fb5e810ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.796+0000 7fb5eff94700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e810f2b0 con 0x7fb5e80fcd60 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5edd30700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 0x7fb5e810ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5edd30700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 0x7fb5e810ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33646/0 (socket says 172.21.15.59:33646) 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5edd30700 1 -- 172.21.15.59:0/2643906765 learned_addr learned my addr 172.21.15.59:0/2643906765 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5edd30700 1 -- 172.21.15.59:0/2643906765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4005040 con 0x7fb5e80fcd60 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5edd30700 1 --2- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 0x7fb5e810ed70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009680 tx=0x7fb5e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5de7fc700 1 -- 172.21.15.59:0/2643906765 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e400b5c0 con 0x7fb5e80fcd60 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e810f4b0 con 0x7fb5e80fcd60 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5de7fc700 1 -- 172.21.15.59:0/2643906765 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5e4004e60 con 0x7fb5e80fcd60 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5de7fc700 1 -- 172.21.15.59:0/2643906765 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5e401c5b0 con 0x7fb5e80fcd60 2023-12-19T18:40:24.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.797+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e8110070 con 0x7fb5e80fcd60 2023-12-19T18:40:24.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.798+0000 7fb5de7fc700 1 -- 172.21.15.59:0/2643906765 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fb5e4013420 con 0x7fb5e80fcd60 2023-12-19T18:40:24.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.798+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e805f270 con 0x7fb5e80fcd60 2023-12-19T18:40:24.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.798+0000 7fb5de7fc700 1 --2- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb5d40366d0 0x7fb5d4038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:24.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.798+0000 7fb5de7fc700 1 -- 172.21.15.59:0/2643906765 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb5e4049780 con 0x7fb5e80fcd60 2023-12-19T18:40:24.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.799+0000 7fb5ed52f700 1 --2- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb5d40366d0 0x7fb5d4038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:24.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.799+0000 7fb5ed52f700 1 --2- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb5d40366d0 0x7fb5d4038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009990 tx=0x7fb5d8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:24.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:24.803+0000 7fb5de7fc700 1 -- 172.21.15.59:0/2643906765 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb5e40119c0 con 0x7fb5e80fcd60 2023-12-19T18:40:25.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.003+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 --> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb5e8110cb0 con 0x7fb5d40366d0 2023-12-19T18:40:25.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.004+0000 7fb5de7fc700 1 -- 172.21.15.59:0/2643906765 <== mgr.14160 v2:172.21.15.59:6800/6145510 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7fb5e8110cb0 con 0x7fb5d40366d0 2023-12-19T18:40:25.004 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:40:25.004 INFO:teuthology.orchestra.run.smithi059.stdout:[{"addr": "172.21.15.59", "hostname": "smithi059", "labels": [], "status": ""}, {"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}] 2023-12-19T18:40:25.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.006+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb5d40366d0 msgr2=0x7fb5d4038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:25.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.006+0000 7fb5eff94700 1 --2- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb5d40366d0 0x7fb5d4038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009990 tx=0x7fb5d8006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:25.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.006+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 msgr2=0x7fb5e810ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:25.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.007+0000 7fb5eff94700 1 --2- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 0x7fb5e810ed70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009680 tx=0x7fb5e4009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:25.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.007+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 shutdown_connections 2023-12-19T18:40:25.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.007+0000 7fb5eff94700 1 --2- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb5d40366d0 0x7fb5d4038b80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:25.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.007+0000 7fb5eff94700 1 --2- 172.21.15.59:0/2643906765 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5e80fcd60 0x7fb5e810ed70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:25.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.007+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 >> 172.21.15.59:0/2643906765 conn(0x7fb5e80f65d0 msgr2=0x7fb5e80f73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:25.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.007+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 shutdown_connections 2023-12-19T18:40:25.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:25.007+0000 7fb5eff94700 1 -- 172.21.15.59:0/2643906765 wait complete. 2023-12-19T18:40:25.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:25 smithi059 ceph-mon[80661]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:26.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:26 smithi059 ceph-mon[80661]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T18:40:27.022 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-19T18:40:27.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd crush tunables default 2023-12-19T18:40:28.046 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:40:28.136 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:27 smithi059 ceph-mon[80661]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:30.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:29 smithi059 ceph-mon[80661]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:30.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:29 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:30.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:29 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:30.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:29 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:30.140 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:29 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.139+0000 7f7a6d42f700 1 -- 172.21.15.59:0/1799360062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 msgr2=0x7f7a680fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.139+0000 7f7a6d42f700 1 --2- 172.21.15.59:0/1799360062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 0x7f7a680fbb80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7a580048c0 tx=0x7f7a58009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.140+0000 7f7a6d42f700 1 -- 172.21.15.59:0/1799360062 shutdown_connections 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.140+0000 7f7a6d42f700 1 --2- 172.21.15.59:0/1799360062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 0x7f7a680fbb80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.140+0000 7f7a6d42f700 1 -- 172.21.15.59:0/1799360062 >> 172.21.15.59:0/1799360062 conn(0x7f7a680f6e50 msgr2=0x7f7a680f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.140+0000 7f7a6d42f700 1 -- 172.21.15.59:0/1799360062 shutdown_connections 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.140+0000 7f7a6d42f700 1 -- 172.21.15.59:0/1799360062 wait complete. 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.140+0000 7f7a6d42f700 1 Processor -- start 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.140+0000 7f7a6d42f700 1 -- start start 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.141+0000 7f7a6d42f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 0x7f7a6810cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.141+0000 7f7a6d42f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6810d000 con 0x7f7a680fb770 2023-12-19T18:40:30.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.141+0000 7f7a66ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 0x7f7a6810cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.141+0000 7f7a66ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 0x7f7a6810cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33666/0 (socket says 172.21.15.59:33666) 2023-12-19T18:40:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.141+0000 7f7a66ffd700 1 -- 172.21.15.59:0/2417141018 learned_addr learned my addr 172.21.15.59:0/2417141018 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:40:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.141+0000 7f7a66ffd700 1 -- 172.21.15.59:0/2417141018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a58005040 con 0x7f7a680fb770 2023-12-19T18:40:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.141+0000 7f7a66ffd700 1 --2- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 0x7f7a6810cac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7a5800b690 tx=0x7f7a58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.142+0000 7f7a4f7fe700 1 -- 172.21.15.59:0/2417141018 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a58009650 con 0x7f7a680fb770 2023-12-19T18:40:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.142+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6810d200 con 0x7f7a680fb770 2023-12-19T18:40:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.142+0000 7f7a4f7fe700 1 -- 172.21.15.59:0/2417141018 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a580097b0 con 0x7f7a680fb770 2023-12-19T18:40:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.142+0000 7f7a4f7fe700 1 -- 172.21.15.59:0/2417141018 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5801c5d0 con 0x7f7a680fb770 2023-12-19T18:40:30.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.142+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6810dea0 con 0x7f7a680fb770 2023-12-19T18:40:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.143+0000 7f7a4f7fe700 1 -- 172.21.15.59:0/2417141018 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f7a5801c840 con 0x7f7a680fb770 2023-12-19T18:40:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.143+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6805f270 con 0x7f7a680fb770 2023-12-19T18:40:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.143+0000 7f7a4f7fe700 1 --2- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f7a5003aae0 0x7f7a5003cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.143+0000 7f7a4f7fe700 1 -- 172.21.15.59:0/2417141018 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7a58049b10 con 0x7f7a680fb770 2023-12-19T18:40:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.143+0000 7f7a667fc700 1 --2- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f7a5003aae0 0x7f7a5003cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:30.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.144+0000 7f7a667fc700 1 --2- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f7a5003aae0 0x7f7a5003cf90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009990 tx=0x7f7a5c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:30.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.148+0000 7f7a4f7fe700 1 -- 172.21.15.59:0/2417141018 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7a58023950 con 0x7f7a680fb770 2023-12-19T18:40:30.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.395+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7a6804bb80 con 0x7f7a680fb770 2023-12-19T18:40:30.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.927+0000 7f7a4f7fe700 1 -- 172.21.15.59:0/2417141018 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7a58023300 con 0x7f7a680fb770 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f7a5003aae0 msgr2=0x7f7a5003cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 --2- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f7a5003aae0 0x7f7a5003cf90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009990 tx=0x7f7a5c006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 msgr2=0x7f7a6810cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 --2- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 0x7f7a6810cac0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7a5800b690 tx=0x7f7a58009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 shutdown_connections 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 --2- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f7a5003aae0 0x7f7a5003cf90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 --2- 172.21.15.59:0/2417141018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7a680fb770 0x7f7a6810cac0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 >> 172.21.15.59:0/2417141018 conn(0x7f7a680f6e50 msgr2=0x7f7a680660a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 shutdown_connections 2023-12-19T18:40:30.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:40:30.930+0000 7f7a6d42f700 1 -- 172.21.15.59:0/2417141018 wait complete. 2023-12-19T18:40:30.931 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2023-12-19T18:40:31.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:30 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:31.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:30 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:31.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:30 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-19T18:40:31.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:30 smithi059 ceph-mon[80661]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-19T18:40:31.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:30 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:31.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:30 smithi059 ceph-mon[80661]: Deploying daemon grafana.smithi059 on smithi059 2023-12-19T18:40:31.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:30 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2417141018' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-19T18:40:31.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:30 smithi059 ceph-mon[80661]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:31.580 INFO:tasks.cephadm:Adding mon.smithi059 on smithi059 2023-12-19T18:40:31.580 INFO:tasks.cephadm:Adding mon.smithi089 on smithi089 2023-12-19T18:40:31.580 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch apply mon '2;smithi059:172.21.15.59=smithi059;smithi089:172.21.15.89=smithi089' 2023-12-19T18:40:31.827 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:32.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:31 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2417141018' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-19T18:40:32.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:31 smithi059 ceph-mon[80661]: osdmap e4: 0 total, 0 up, 0 in 2023-12-19T18:40:33.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:32 smithi059 ceph-mon[80661]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.350+0000 7f2580424700 1 -- 172.21.15.89:0/1605422038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 msgr2=0x7f25780ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.350+0000 7f2580424700 1 --2- 172.21.15.89:0/1605422038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 0x7f25780ff7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2568009c60 tx=0x7f2568004d10 comp rx=0 tx=0).stop 2023-12-19T18:40:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.350+0000 7f2580424700 1 -- 172.21.15.89:0/1605422038 shutdown_connections 2023-12-19T18:40:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.350+0000 7f2580424700 1 --2- 172.21.15.89:0/1605422038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 0x7f25780ff7a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.350+0000 7f2580424700 1 -- 172.21.15.89:0/1605422038 >> 172.21.15.89:0/1605422038 conn(0x7f25780f6e40 msgr2=0x7f25780f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.350+0000 7f2580424700 1 -- 172.21.15.89:0/1605422038 shutdown_connections 2023-12-19T18:40:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.350+0000 7f2580424700 1 -- 172.21.15.89:0/1605422038 wait complete. 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.351+0000 7f2580424700 1 Processor -- start 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.351+0000 7f2580424700 1 -- start start 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.351+0000 7f2580424700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 0x7f25780fce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.351+0000 7f2580424700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25780fb2d0 con 0x7f25780fd3c0 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.351+0000 7f257e1c0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 0x7f25780fce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.351+0000 7f257e1c0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 0x7f25780fce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:57810/0 (socket says 172.21.15.89:57810) 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.351+0000 7f257e1c0700 1 -- 172.21.15.89:0/2035636358 learned_addr learned my addr 172.21.15.89:0/2035636358 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.351+0000 7f257e1c0700 1 -- 172.21.15.89:0/2035636358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2568009940 con 0x7f25780fd3c0 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.352+0000 7f257e1c0700 1 --2- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 0x7f25780fce50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2568007b30 tx=0x7f2568004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.352+0000 7f2576ffd700 1 -- 172.21.15.89:0/2035636358 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f256800bec0 con 0x7f25780fd3c0 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.352+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25780fb4d0 con 0x7f25780fd3c0 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.352+0000 7f2576ffd700 1 -- 172.21.15.89:0/2035636358 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f256801d8f0 con 0x7f25780fd3c0 2023-12-19T18:40:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.352+0000 7f2576ffd700 1 -- 172.21.15.89:0/2035636358 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25680128e0 con 0x7f25780fd3c0 2023-12-19T18:40:33.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.352+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25780fb9c0 con 0x7f25780fd3c0 2023-12-19T18:40:33.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.353+0000 7f2576ffd700 1 -- 172.21.15.89:0/2035636358 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f2568019040 con 0x7f25780fd3c0 2023-12-19T18:40:33.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.353+0000 7f2576ffd700 1 --2- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f25600366d0 0x7f2560038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:33.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.353+0000 7f2576ffd700 1 -- 172.21.15.89:0/2035636358 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f256804a360 con 0x7f25780fd3c0 2023-12-19T18:40:33.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.353+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2578106510 con 0x7f25780fd3c0 2023-12-19T18:40:33.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.353+0000 7f257d9bf700 1 --2- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f25600366d0 0x7f2560038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:33.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.353+0000 7f257d9bf700 1 --2- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f25600366d0 0x7f2560038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f256c009990 tx=0x7f256c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:33.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.359+0000 7f2576ffd700 1 -- 172.21.15.89:0/2035636358 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2568021020 con 0x7f25780fd3c0 2023-12-19T18:40:33.552 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.553+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 --> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi089:172.21.15.89=smithi089", "target": ["mon-mgr", ""]}) v1 -- 0x7f257802b200 con 0x7f25600366d0 2023-12-19T18:40:33.558 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.558+0000 7f2576ffd700 1 -- 172.21.15.89:0/2035636358 <== mgr.14160 v2:172.21.15.59:6800/6145510 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f257802b200 con 0x7f25600366d0 2023-12-19T18:40:33.558 INFO:teuthology.orchestra.run.smithi089.stdout:Scheduled mon update... 2023-12-19T18:40:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f25600366d0 msgr2=0x7f2560038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 --2- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f25600366d0 0x7f2560038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f256c009990 tx=0x7f256c006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 msgr2=0x7f25780fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 --2- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 0x7f25780fce50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2568007b30 tx=0x7f2568004d10 comp rx=0 tx=0).stop 2023-12-19T18:40:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 shutdown_connections 2023-12-19T18:40:33.560 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 --2- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f25600366d0 0x7f2560038b80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:33.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 --2- 172.21.15.89:0/2035636358 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25780fd3c0 0x7f25780fce50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:33.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 >> 172.21.15.89:0/2035636358 conn(0x7f25780f6e40 msgr2=0x7f25780f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:33.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 shutdown_connections 2023-12-19T18:40:33.561 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:33.560+0000 7f2580424700 1 -- 172.21.15.89:0/2035636358 wait complete. 2023-12-19T18:40:34.113 DEBUG:teuthology.orchestra.run.smithi089:mon.smithi089> sudo journalctl -f -n 0 -u ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mon.smithi089.service 2023-12-19T18:40:34.115 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:40:34.116 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:40:34.389 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:34.435 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:34.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:34 smithi059 ceph-mon[80661]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi089:172.21.15.89=smithi089", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:40:34.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:34 smithi059 ceph-mon[80661]: Saving service mon spec with placement smithi059:172.21.15.59=smithi059;smithi089:172.21.15.89=smithi089;count:2 2023-12-19T18:40:34.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:34 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:34.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:34 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:35.730 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.730+0000 7f2d9255f700 1 -- 172.21.15.89:0/1345115490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 msgr2=0x7f2d8c0ff420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:35.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.730+0000 7f2d9255f700 1 --2- 172.21.15.89:0/1345115490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 0x7f2d8c0ff420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0048c0 tx=0x7f2d7c009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:35.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.730+0000 7f2d9255f700 1 -- 172.21.15.89:0/1345115490 shutdown_connections 2023-12-19T18:40:35.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.730+0000 7f2d9255f700 1 --2- 172.21.15.89:0/1345115490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 0x7f2d8c0ff420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:35.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.730+0000 7f2d9255f700 1 -- 172.21.15.89:0/1345115490 >> 172.21.15.89:0/1345115490 conn(0x7f2d8c0f6e50 msgr2=0x7f2d8c0f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:35.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.730+0000 7f2d9255f700 1 -- 172.21.15.89:0/1345115490 shutdown_connections 2023-12-19T18:40:35.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.730+0000 7f2d9255f700 1 -- 172.21.15.89:0/1345115490 wait complete. 2023-12-19T18:40:35.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.731+0000 7f2d9255f700 1 Processor -- start 2023-12-19T18:40:35.731 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.731+0000 7f2d9255f700 1 -- start start 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.731+0000 7f2d9255f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 0x7f2d8c0fcea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.731+0000 7f2d9255f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0fb320 con 0x7f2d8c066530 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.731+0000 7f2d8bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 0x7f2d8c0fcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.731+0000 7f2d8bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 0x7f2d8c0fcea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:57830/0 (socket says 172.21.15.89:57830) 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.731+0000 7f2d8bfff700 1 -- 172.21.15.89:0/1358295199 learned_addr learned my addr 172.21.15.89:0/1358295199 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.732+0000 7f2d8bfff700 1 -- 172.21.15.89:0/1358295199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c005040 con 0x7f2d8c066530 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.732+0000 7f2d8bfff700 1 --2- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 0x7f2d8c0fcea0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009040 tx=0x7f2d7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.732+0000 7f2d88ff9700 1 -- 172.21.15.89:0/1358295199 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d7c009620 con 0x7f2d8c066530 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.732+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c0fb520 con 0x7f2d8c066530 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.732+0000 7f2d88ff9700 1 -- 172.21.15.89:0/1358295199 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d7c009780 con 0x7f2d8c066530 2023-12-19T18:40:35.733 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.732+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c0fba10 con 0x7f2d8c066530 2023-12-19T18:40:35.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.732+0000 7f2d88ff9700 1 -- 172.21.15.89:0/1358295199 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d7c025560 con 0x7f2d8c066530 2023-12-19T18:40:35.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.733+0000 7f2d88ff9700 1 -- 172.21.15.89:0/1358295199 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f2d7c013a60 con 0x7f2d8c066530 2023-12-19T18:40:35.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.733+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c1064f0 con 0x7f2d8c066530 2023-12-19T18:40:35.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.733+0000 7f2d88ff9700 1 --2- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2d740366d0 0x7f2d74038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:35.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.733+0000 7f2d88ff9700 1 -- 172.21.15.89:0/1358295199 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2d7c049900 con 0x7f2d8c066530 2023-12-19T18:40:35.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.733+0000 7f2d8b7fe700 1 --2- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2d740366d0 0x7f2d74038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:35.734 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.734+0000 7f2d8b7fe700 1 --2- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2d740366d0 0x7f2d74038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009990 tx=0x7f2d80006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:35.741 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:35.742+0000 7f2d88ff9700 1 -- 172.21.15.89:0/1358295199 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2d7c011b60 con 0x7f2d8c066530 2023-12-19T18:40:35.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:35 smithi059 ceph-mon[80661]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.007+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d8c05f270 con 0x7f2d8c066530 2023-12-19T18:40:36.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.007+0000 7f2d88ff9700 1 -- 172.21.15.89:0/1358295199 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2d7c011b60 con 0x7f2d8c066530 2023-12-19T18:40:36.007 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:40:36.007 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:40:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2d740366d0 msgr2=0x7f2d74038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 --2- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2d740366d0 0x7f2d74038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009990 tx=0x7f2d80006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 msgr2=0x7f2d8c0fcea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 --2- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 0x7f2d8c0fcea0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009040 tx=0x7f2d7c009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 shutdown_connections 2023-12-19T18:40:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 --2- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2d740366d0 0x7f2d74038b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 --2- 172.21.15.89:0/1358295199 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d8c066530 0x7f2d8c0fcea0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:36.009 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 >> 172.21.15.89:0/1358295199 conn(0x7f2d8c0f6e50 msgr2=0x7f2d8c0f7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:36.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 shutdown_connections 2023-12-19T18:40:36.010 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:36.009+0000 7f2d9255f700 1 -- 172.21.15.89:0/1358295199 wait complete. 2023-12-19T18:40:36.010 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:40:36.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:36 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/1358295199' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:40:37.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:40:37.441 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:40:37.689 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:37.734 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:38.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:37 smithi059 ceph-mon[80661]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:39.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.175+0000 7fd6205de700 1 -- 172.21.15.89:0/2767817961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 msgr2=0x7fd6180ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:39.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.175+0000 7fd6205de700 1 --2- 172.21.15.89:0/2767817961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 0x7fd6180ff450 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd6080048c0 tx=0x7fd608009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:39.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.175+0000 7fd6205de700 1 -- 172.21.15.89:0/2767817961 shutdown_connections 2023-12-19T18:40:39.176 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.175+0000 7fd6205de700 1 --2- 172.21.15.89:0/2767817961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 0x7fd6180ff450 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.175+0000 7fd6205de700 1 -- 172.21.15.89:0/2767817961 >> 172.21.15.89:0/2767817961 conn(0x7fd6180f6e40 msgr2=0x7fd6180f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.175+0000 7fd6205de700 1 -- 172.21.15.89:0/2767817961 shutdown_connections 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.175+0000 7fd6205de700 1 -- 172.21.15.89:0/2767817961 wait complete. 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.176+0000 7fd6205de700 1 Processor -- start 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.176+0000 7fd6205de700 1 -- start start 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.176+0000 7fd6205de700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 0x7fd61810ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.176+0000 7fd6205de700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61810cfa0 con 0x7fd618065a80 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.176+0000 7fd61e37a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 0x7fd61810ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.176+0000 7fd61e37a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 0x7fd61810ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:57852/0 (socket says 172.21.15.89:57852) 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.176+0000 7fd61e37a700 1 -- 172.21.15.89:0/4148485176 learned_addr learned my addr 172.21.15.89:0/4148485176 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.176+0000 7fd61e37a700 1 -- 172.21.15.89:0/4148485176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd608005040 con 0x7fd618065a80 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.176+0000 7fd61e37a700 1 --2- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 0x7fd61810ca60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd60800b690 tx=0x7fd608009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.177+0000 7fd60effd700 1 -- 172.21.15.89:0/4148485176 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd608009650 con 0x7fd618065a80 2023-12-19T18:40:39.177 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.177+0000 7fd60effd700 1 -- 172.21.15.89:0/4148485176 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6080097b0 con 0x7fd618065a80 2023-12-19T18:40:39.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.177+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61810d1a0 con 0x7fd618065a80 2023-12-19T18:40:39.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.177+0000 7fd60effd700 1 -- 172.21.15.89:0/4148485176 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd60801c5b0 con 0x7fd618065a80 2023-12-19T18:40:39.178 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.177+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61810de40 con 0x7fd618065a80 2023-12-19T18:40:39.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.178+0000 7fd60effd700 1 -- 172.21.15.89:0/4148485176 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fd608013420 con 0x7fd618065a80 2023-12-19T18:40:39.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.178+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61805f270 con 0x7fd618065a80 2023-12-19T18:40:39.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.178+0000 7fd60effd700 1 --2- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fd6040366d0 0x7fd604038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:39.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.178+0000 7fd60effd700 1 -- 172.21.15.89:0/4148485176 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd608049910 con 0x7fd618065a80 2023-12-19T18:40:39.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.178+0000 7fd61db79700 1 --2- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fd6040366d0 0x7fd604038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:39.179 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.178+0000 7fd61db79700 1 --2- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fd6040366d0 0x7fd604038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd614009990 tx=0x7fd614006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:39.182 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.183+0000 7fd60effd700 1 -- 172.21.15.89:0/4148485176 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd608013a60 con 0x7fd618065a80 2023-12-19T18:40:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.456+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd61810e9b0 con 0x7fd618065a80 2023-12-19T18:40:39.456 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.457+0000 7fd60effd700 1 -- 172.21.15.89:0/4148485176 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd608011370 con 0x7fd618065a80 2023-12-19T18:40:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:40:39.457 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:40:39.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fd6040366d0 msgr2=0x7fd604038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:39.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 --2- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fd6040366d0 0x7fd604038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd614009990 tx=0x7fd614006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:39.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 msgr2=0x7fd61810ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:39.458 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 --2- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 0x7fd61810ca60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd60800b690 tx=0x7fd608009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 shutdown_connections 2023-12-19T18:40:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 --2- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fd6040366d0 0x7fd604038b80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 --2- 172.21.15.89:0/4148485176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd618065a80 0x7fd61810ca60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 >> 172.21.15.89:0/4148485176 conn(0x7fd6180f6e40 msgr2=0x7fd6180f7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 shutdown_connections 2023-12-19T18:40:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:39.459+0000 7fd6205de700 1 -- 172.21.15.89:0/4148485176 wait complete. 2023-12-19T18:40:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:40:40.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:39 smithi059 ceph-mon[80661]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:40.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:39 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/4148485176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:40:40.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:40:40.955 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:40:41.207 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:41.253 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:41.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:41 smithi059 ceph-mon[80661]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:42.675 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.675+0000 7f300b9cd700 1 -- 172.21.15.89:0/3862076297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 msgr2=0x7f30040fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:42.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.675+0000 7f300b9cd700 1 --2- 172.21.15.89:0/3862076297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 0x7f30040fcc90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40048c0 tx=0x7f2ff4009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:42.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.675+0000 7f300b9cd700 1 -- 172.21.15.89:0/3862076297 shutdown_connections 2023-12-19T18:40:42.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.675+0000 7f300b9cd700 1 --2- 172.21.15.89:0/3862076297 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 0x7f30040fcc90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:42.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.675+0000 7f300b9cd700 1 -- 172.21.15.89:0/3862076297 >> 172.21.15.89:0/3862076297 conn(0x7f30040f7f40 msgr2=0x7f30040fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:42.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.675+0000 7f300b9cd700 1 -- 172.21.15.89:0/3862076297 shutdown_connections 2023-12-19T18:40:42.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.675+0000 7f300b9cd700 1 -- 172.21.15.89:0/3862076297 wait complete. 2023-12-19T18:40:42.703 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.676+0000 7f300b9cd700 1 Processor -- start 2023-12-19T18:40:42.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.676+0000 7f300b9cd700 1 -- start start 2023-12-19T18:40:42.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.676+0000 7f300b9cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 0x7f300410cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:42.704 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.676+0000 7f300b9cd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300410d000 con 0x7f30040fc880 2023-12-19T18:40:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.676+0000 7f3009769700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 0x7f300410cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.676+0000 7f3009769700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 0x7f300410cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:36328/0 (socket says 172.21.15.89:36328) 2023-12-19T18:40:42.706 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.676+0000 7f3009769700 1 -- 172.21.15.89:0/912309668 learned_addr learned my addr 172.21.15.89:0/912309668 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.677+0000 7f3009769700 1 -- 172.21.15.89:0/912309668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4005040 con 0x7f30040fc880 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.677+0000 7f3009769700 1 --2- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 0x7f300410cac0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009330 tx=0x7f2ff4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.677+0000 7f2ff9ffb700 1 -- 172.21.15.89:0/912309668 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff40096b0 con 0x7f30040fc880 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.677+0000 7f2ff9ffb700 1 -- 172.21.15.89:0/912309668 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ff4009810 con 0x7f30040fc880 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.677+0000 7f2ff9ffb700 1 -- 172.21.15.89:0/912309668 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ff401c5b0 con 0x7f30040fc880 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.677+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300410d200 con 0x7f30040fc880 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.677+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300410dea0 con 0x7f30040fc880 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.678+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300405f270 con 0x7f30040fc880 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.678+0000 7f2ff9ffb700 1 -- 172.21.15.89:0/912309668 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f2ff4013420 con 0x7f30040fc880 2023-12-19T18:40:42.707 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.678+0000 7f2ff9ffb700 1 --2- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2ff00366d0 0x7f2ff0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.678+0000 7f2ff9ffb700 1 -- 172.21.15.89:0/912309668 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2ff4049a90 con 0x7f30040fc880 2023-12-19T18:40:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.679+0000 7f3008f68700 1 --2- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2ff00366d0 0x7f2ff0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.679+0000 7f3008f68700 1 --2- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2ff00366d0 0x7f2ff0038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3000009990 tx=0x7f3000006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:42.708 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.684+0000 7f2ff9ffb700 1 -- 172.21.15.89:0/912309668 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2ff40119c0 con 0x7f30040fc880 2023-12-19T18:40:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.940+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f300410e9f0 con 0x7f30040fc880 2023-12-19T18:40:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.941+0000 7f2ff9ffb700 1 -- 172.21.15.89:0/912309668 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2ff4011370 con 0x7f30040fc880 2023-12-19T18:40:42.940 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:40:42.940 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:40:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2ff00366d0 msgr2=0x7f2ff0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 --2- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2ff00366d0 0x7f2ff0038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3000009990 tx=0x7f3000006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 msgr2=0x7f300410cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 --2- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 0x7f300410cac0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009330 tx=0x7f2ff4009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 shutdown_connections 2023-12-19T18:40:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 --2- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f2ff00366d0 0x7f2ff0038b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 --2- 172.21.15.89:0/912309668 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f30040fc880 0x7f300410cac0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 >> 172.21.15.89:0/912309668 conn(0x7f30040f7f40 msgr2=0x7f30040f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 shutdown_connections 2023-12-19T18:40:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:42.943+0000 7f300b9cd700 1 -- 172.21.15.89:0/912309668 wait complete. 2023-12-19T18:40:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:40:44.085 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:43 smithi059 ceph-mon[80661]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:44.085 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:43 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/912309668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:40:44.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:40:44.451 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:44.744 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:44 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:44 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:44 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:44 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:44 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:44 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:44 smithi059 ceph-mon[80661]: Deploying daemon prometheus.smithi059 on smithi059 2023-12-19T18:40:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:44 smithi059 ceph-mon[80661]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:45.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.956+0000 7f8a11b81700 1 -- 172.21.15.89:0/1505797452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 msgr2=0x7f8a0c0fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:45.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.956+0000 7f8a11b81700 1 --2- 172.21.15.89:0/1505797452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 0x7f8a0c0fcc90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0048c0 tx=0x7f89fc009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:45.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.956+0000 7f8a11b81700 1 -- 172.21.15.89:0/1505797452 shutdown_connections 2023-12-19T18:40:45.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.956+0000 7f8a11b81700 1 --2- 172.21.15.89:0/1505797452 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 0x7f8a0c0fcc90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:45.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.956+0000 7f8a11b81700 1 -- 172.21.15.89:0/1505797452 >> 172.21.15.89:0/1505797452 conn(0x7f8a0c0f7f40 msgr2=0x7f8a0c0fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:45.957 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.956+0000 7f8a11b81700 1 -- 172.21.15.89:0/1505797452 shutdown_connections 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.956+0000 7f8a11b81700 1 -- 172.21.15.89:0/1505797452 wait complete. 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.957+0000 7f8a11b81700 1 Processor -- start 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.957+0000 7f8a11b81700 1 -- start start 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.957+0000 7f8a11b81700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 0x7f8a0c10cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.957+0000 7f8a11b81700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c10d000 con 0x7f8a0c0fc880 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.957+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 0x7f8a0c10cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.957+0000 7f8a0b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 0x7f8a0c10cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:36338/0 (socket says 172.21.15.89:36338) 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.957+0000 7f8a0b7fe700 1 -- 172.21.15.89:0/693351300 learned_addr learned my addr 172.21.15.89:0/693351300 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.958+0000 7f8a0b7fe700 1 -- 172.21.15.89:0/693351300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc005040 con 0x7f8a0c0fc880 2023-12-19T18:40:45.958 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.958+0000 7f8a0b7fe700 1 --2- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 0x7f8a0c10cac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009330 tx=0x7f89fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:45.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.958+0000 7f89f3fff700 1 -- 172.21.15.89:0/693351300 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89fc0096b0 con 0x7f8a0c0fc880 2023-12-19T18:40:45.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.958+0000 7f89f3fff700 1 -- 172.21.15.89:0/693351300 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89fc009810 con 0x7f8a0c0fc880 2023-12-19T18:40:45.959 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.958+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0c10d200 con 0x7f8a0c0fc880 2023-12-19T18:40:45.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.958+0000 7f89f3fff700 1 -- 172.21.15.89:0/693351300 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89fc01c5b0 con 0x7f8a0c0fc880 2023-12-19T18:40:45.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.958+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0c10dea0 con 0x7f8a0c0fc880 2023-12-19T18:40:45.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.959+0000 7f89f3fff700 1 -- 172.21.15.89:0/693351300 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f89fc013420 con 0x7f8a0c0fc880 2023-12-19T18:40:45.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.959+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0c05f270 con 0x7f8a0c0fc880 2023-12-19T18:40:45.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.959+0000 7f89f3fff700 1 --2- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f89f4036720 0x7f89f4038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:45.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.959+0000 7f89f3fff700 1 -- 172.21.15.89:0/693351300 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f89fc049a70 con 0x7f8a0c0fc880 2023-12-19T18:40:45.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.959+0000 7f8a0affd700 1 --2- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f89f4036720 0x7f89f4038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:45.960 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.959+0000 7f8a0affd700 1 --2- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f89f4036720 0x7f89f4038bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009990 tx=0x7f8a00006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:45.963 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:45.964+0000 7f89f3fff700 1 -- 172.21.15.89:0/693351300 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f89fc0119c0 con 0x7f8a0c0fc880 2023-12-19T18:40:46.220 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.220+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a0c10e9f0 con 0x7f8a0c0fc880 2023-12-19T18:40:46.221 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.221+0000 7f89f3fff700 1 -- 172.21.15.89:0/693351300 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f89fc011370 con 0x7f8a0c0fc880 2023-12-19T18:40:46.221 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:40:46.221 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:40:46.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.223+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f89f4036720 msgr2=0x7f89f4038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:46.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.223+0000 7f8a11b81700 1 --2- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f89f4036720 0x7f89f4038bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009990 tx=0x7f8a00006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:46.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.223+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 msgr2=0x7f8a0c10cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:46.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.223+0000 7f8a11b81700 1 --2- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 0x7f8a0c10cac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009330 tx=0x7f89fc009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:46.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.223+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 shutdown_connections 2023-12-19T18:40:46.223 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.223+0000 7f8a11b81700 1 --2- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f89f4036720 0x7f89f4038bd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:46.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.223+0000 7f8a11b81700 1 --2- 172.21.15.89:0/693351300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a0c0fc880 0x7f8a0c10cac0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:46.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.223+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 >> 172.21.15.89:0/693351300 conn(0x7f8a0c0f7f40 msgr2=0x7f8a0c0f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:46.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.223+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 shutdown_connections 2023-12-19T18:40:46.224 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:46.224+0000 7f8a11b81700 1 -- 172.21.15.89:0/693351300 wait complete. 2023-12-19T18:40:46.224 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:40:46.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:46 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/693351300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:40:47.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:40:47.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:40:48.045 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:48.091 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:48.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:47 smithi059 ceph-mon[80661]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:49.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.583+0000 7f8663d58700 1 -- 172.21.15.89:0/2645630626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 msgr2=0x7f865c0fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:49.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.583+0000 7f8663d58700 1 --2- 172.21.15.89:0/2645630626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 0x7f865c0fbb90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f86580048c0 tx=0x7f8658009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:49.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.583+0000 7f8663d58700 1 -- 172.21.15.89:0/2645630626 shutdown_connections 2023-12-19T18:40:49.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.583+0000 7f8663d58700 1 --2- 172.21.15.89:0/2645630626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 0x7f865c0fbb90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:49.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.583+0000 7f8663d58700 1 -- 172.21.15.89:0/2645630626 >> 172.21.15.89:0/2645630626 conn(0x7f865c0f6e40 msgr2=0x7f865c0f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:49.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.583+0000 7f8663d58700 1 -- 172.21.15.89:0/2645630626 shutdown_connections 2023-12-19T18:40:49.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.583+0000 7f8663d58700 1 -- 172.21.15.89:0/2645630626 wait complete. 2023-12-19T18:40:49.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.583+0000 7f8663d58700 1 Processor -- start 2023-12-19T18:40:49.583 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.583+0000 7f8663d58700 1 -- start start 2023-12-19T18:40:49.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f8663d58700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 0x7f865c10a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:49.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f8663d58700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c10ae10 con 0x7f865c0fb780 2023-12-19T18:40:49.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f8661af4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 0x7f865c10a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:49.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f8661af4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 0x7f865c10a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:36350/0 (socket says 172.21.15.89:36350) 2023-12-19T18:40:49.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f8661af4700 1 -- 172.21.15.89:0/598160529 learned_addr learned my addr 172.21.15.89:0/598160529 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:40:49.585 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f8661af4700 1 -- 172.21.15.89:0/598160529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8658005040 con 0x7f865c0fb780 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f8661af4700 1 --2- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 0x7f865c10a8d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f865800b690 tx=0x7f8658009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f86527fc700 1 -- 172.21.15.89:0/598160529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8658009650 con 0x7f865c0fb780 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865c10b010 con 0x7f865c0fb780 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f86527fc700 1 -- 172.21.15.89:0/598160529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86580097b0 con 0x7f865c0fb780 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f86527fc700 1 -- 172.21.15.89:0/598160529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8658025560 con 0x7f865c0fb780 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.584+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865c10bcb0 con 0x7f865c0fb780 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.585+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865c05f270 con 0x7f865c0fb780 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.585+0000 7f86527fc700 1 -- 172.21.15.89:0/598160529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f86580139d0 con 0x7f865c0fb780 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.585+0000 7f86527fc700 1 --2- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f8648038930 0x7f864803ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.585+0000 7f86527fc700 1 -- 172.21.15.89:0/598160529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f86580499e0 con 0x7f865c0fb780 2023-12-19T18:40:49.586 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.586+0000 7f86612f3700 1 --2- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f8648038930 0x7f864803ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:49.587 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.586+0000 7f86612f3700 1 --2- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f8648038930 0x7f864803ade0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f864c009990 tx=0x7f864c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:49.590 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.590+0000 7f86527fc700 1 -- 172.21.15.89:0/598160529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f86580256c0 con 0x7f865c0fb780 2023-12-19T18:40:49.846 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.847+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f865c10c800 con 0x7f865c0fb780 2023-12-19T18:40:49.847 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.847+0000 7f86527fc700 1 -- 172.21.15.89:0/598160529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f865801be60 con 0x7f865c0fb780 2023-12-19T18:40:49.847 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:40:49.847 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:40:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f8648038930 msgr2=0x7f864803ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 --2- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f8648038930 0x7f864803ade0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f864c009990 tx=0x7f864c006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 msgr2=0x7f865c10a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 --2- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 0x7f865c10a8d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f865800b690 tx=0x7f8658009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 shutdown_connections 2023-12-19T18:40:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 --2- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f8648038930 0x7f864803ade0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 --2- 172.21.15.89:0/598160529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f865c0fb780 0x7f865c10a8d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 >> 172.21.15.89:0/598160529 conn(0x7f865c0f6e40 msgr2=0x7f865c0f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 shutdown_connections 2023-12-19T18:40:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:49.849+0000 7f8663d58700 1 -- 172.21.15.89:0/598160529 wait complete. 2023-12-19T18:40:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:40:50.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:49 smithi059 ceph-mon[80661]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:50.201 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:49 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:40:51.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:50 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/598160529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:40:51.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:40:51.364 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:40:51.604 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:51.651 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:52.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:51 smithi059 ceph-mon[80661]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:52.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.854+0000 7fb9bd739700 1 -- 172.21.15.89:0/2255428852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 msgr2=0x7fb9b80fbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:52.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.854+0000 7fb9bd739700 1 --2- 172.21.15.89:0/2255428852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 0x7fb9b80fbb70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80048c0 tx=0x7fb9a8009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:52.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.854+0000 7fb9bd739700 1 -- 172.21.15.89:0/2255428852 shutdown_connections 2023-12-19T18:40:52.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.854+0000 7fb9bd739700 1 --2- 172.21.15.89:0/2255428852 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 0x7fb9b80fbb70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:52.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.854+0000 7fb9bd739700 1 -- 172.21.15.89:0/2255428852 >> 172.21.15.89:0/2255428852 conn(0x7fb9b80f6ec0 msgr2=0x7fb9b80f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:52.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.854+0000 7fb9bd739700 1 -- 172.21.15.89:0/2255428852 shutdown_connections 2023-12-19T18:40:52.854 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.854+0000 7fb9bd739700 1 -- 172.21.15.89:0/2255428852 wait complete. 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.854+0000 7fb9bd739700 1 Processor -- start 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.854+0000 7fb9bd739700 1 -- start start 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb9bd739700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 0x7fb9b810f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb9bd739700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80664a0 con 0x7fb9b80fb760 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb9b6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 0x7fb9b810f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb9b6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 0x7fb9b810f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:49628/0 (socket says 172.21.15.89:49628) 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb9b6ffd700 1 -- 172.21.15.89:0/2860594051 learned_addr learned my addr 172.21.15.89:0/2860594051 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb9b6ffd700 1 -- 172.21.15.89:0/2860594051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a8005040 con 0x7fb9b80fb760 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb9b6ffd700 1 --2- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 0x7fb9b810f730 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800b690 tx=0x7fb9a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb99f7fe700 1 -- 172.21.15.89:0/2860594051 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9a8009650 con 0x7fb9b80fb760 2023-12-19T18:40:52.856 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b80666a0 con 0x7fb9b80fb760 2023-12-19T18:40:52.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb99f7fe700 1 -- 172.21.15.89:0/2860594051 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9a80097b0 con 0x7fb9b80fb760 2023-12-19T18:40:52.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb99f7fe700 1 -- 172.21.15.89:0/2860594051 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9a801c5b0 con 0x7fb9b80fb760 2023-12-19T18:40:52.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.855+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b810feb0 con 0x7fb9b80fb760 2023-12-19T18:40:52.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.856+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b8105370 con 0x7fb9b80fb760 2023-12-19T18:40:52.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.856+0000 7fb99f7fe700 1 -- 172.21.15.89:0/2860594051 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fb9a8013420 con 0x7fb9b80fb760 2023-12-19T18:40:52.857 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.856+0000 7fb99f7fe700 1 --2- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb9a00366d0 0x7fb9a0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:52.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.857+0000 7fb99f7fe700 1 -- 172.21.15.89:0/2860594051 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb9a80499c0 con 0x7fb9b80fb760 2023-12-19T18:40:52.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.857+0000 7fb9b67fc700 1 --2- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb9a00366d0 0x7fb9a0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:52.858 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.857+0000 7fb9b67fc700 1 --2- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb9a00366d0 0x7fb9a0038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009990 tx=0x7fb9ac006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:52.861 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:52.862+0000 7fb99f7fe700 1 -- 172.21.15.89:0/2860594051 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb9a8025400 con 0x7fb9b80fb760 2023-12-19T18:40:53.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.124+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9b804bb80 con 0x7fb9b80fb760 2023-12-19T18:40:53.124 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.125+0000 7fb99f7fe700 1 -- 172.21.15.89:0/2860594051 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb9a8011370 con 0x7fb9b80fb760 2023-12-19T18:40:53.124 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:40:53.125 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb9a00366d0 msgr2=0x7fb9a0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 --2- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb9a00366d0 0x7fb9a0038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac009990 tx=0x7fb9ac006e40 comp rx=0 tx=0).stop 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 msgr2=0x7fb9b810f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 --2- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 0x7fb9b810f730 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800b690 tx=0x7fb9a8009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 shutdown_connections 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 --2- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7fb9a00366d0 0x7fb9a0038b80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 --2- 172.21.15.89:0/2860594051 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9b80fb760 0x7fb9b810f730 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 >> 172.21.15.89:0/2860594051 conn(0x7fb9b80f6ec0 msgr2=0x7fb9b80f7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 shutdown_connections 2023-12-19T18:40:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:53.127+0000 7fb9bd739700 1 -- 172.21.15.89:0/2860594051 wait complete. 2023-12-19T18:40:53.128 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:40:54.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:53 smithi059 ceph-mon[80661]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:54.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:53 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/2860594051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:40:54.783 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:40:54.783 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:40:55.030 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:55.075 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:56.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:55 smithi059 ceph-mon[80661]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.342+0000 7f17e5cc3700 1 -- 172.21.15.89:0/4294860394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 msgr2=0x7f17e00fbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.342+0000 7f17e5cc3700 1 --2- 172.21.15.89:0/4294860394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 0x7f17e00fbb70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17d80048c0 tx=0x7f17d8009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.342+0000 7f17e5cc3700 1 -- 172.21.15.89:0/4294860394 shutdown_connections 2023-12-19T18:40:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.342+0000 7f17e5cc3700 1 --2- 172.21.15.89:0/4294860394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 0x7f17e00fbb70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.342+0000 7f17e5cc3700 1 -- 172.21.15.89:0/4294860394 >> 172.21.15.89:0/4294860394 conn(0x7f17e00f6ec0 msgr2=0x7f17e00f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.342+0000 7f17e5cc3700 1 -- 172.21.15.89:0/4294860394 shutdown_connections 2023-12-19T18:40:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.342+0000 7f17e5cc3700 1 -- 172.21.15.89:0/4294860394 wait complete. 2023-12-19T18:40:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.343+0000 7f17e5cc3700 1 Processor -- start 2023-12-19T18:40:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.343+0000 7f17e5cc3700 1 -- start start 2023-12-19T18:40:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.343+0000 7f17e5cc3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 0x7f17e00f8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.343+0000 7f17e5cc3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e00f7400 con 0x7f17e00fb760 2023-12-19T18:40:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.343+0000 7f17df7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 0x7f17e00f8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.343+0000 7f17df7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 0x7f17e00f8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:49636/0 (socket says 172.21.15.89:49636) 2023-12-19T18:40:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.343+0000 7f17df7fe700 1 -- 172.21.15.89:0/1077650454 learned_addr learned my addr 172.21.15.89:0/1077650454 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:40:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.343+0000 7f17df7fe700 1 -- 172.21.15.89:0/1077650454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d8005040 con 0x7f17e00fb760 2023-12-19T18:40:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.344+0000 7f17df7fe700 1 --2- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 0x7f17e00f8f80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17d800b690 tx=0x7f17d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.344+0000 7f17c7fff700 1 -- 172.21.15.89:0/1077650454 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17d8009650 con 0x7f17e00fb760 2023-12-19T18:40:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.344+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e00f7600 con 0x7f17e00fb760 2023-12-19T18:40:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.344+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e00f7af0 con 0x7f17e00fb760 2023-12-19T18:40:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.344+0000 7f17c7fff700 1 -- 172.21.15.89:0/1077650454 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17d80097b0 con 0x7f17e00fb760 2023-12-19T18:40:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.344+0000 7f17c7fff700 1 -- 172.21.15.89:0/1077650454 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17d80255c0 con 0x7f17e00fb760 2023-12-19T18:40:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.345+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e0106380 con 0x7f17e00fb760 2023-12-19T18:40:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.345+0000 7f17c7fff700 1 -- 172.21.15.89:0/1077650454 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f17d8013420 con 0x7f17e00fb760 2023-12-19T18:40:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.345+0000 7f17c7fff700 1 --2- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f17c80366d0 0x7f17c8038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.345+0000 7f17c7fff700 1 -- 172.21.15.89:0/1077650454 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f17d80498c0 con 0x7f17e00fb760 2023-12-19T18:40:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.345+0000 7f17deffd700 1 --2- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f17c80366d0 0x7f17c8038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.346+0000 7f17deffd700 1 --2- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f17c80366d0 0x7f17c8038b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f17d400ad90 tx=0x7f17d4009410 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.350+0000 7f17c7fff700 1 -- 172.21.15.89:0/1077650454 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f17d80123c0 con 0x7f17e00fb760 2023-12-19T18:40:56.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.609+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17e0101360 con 0x7f17e00fb760 2023-12-19T18:40:56.609 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.610+0000 7f17c7fff700 1 -- 172.21.15.89:0/1077650454 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f17d8023300 con 0x7f17e00fb760 2023-12-19T18:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f17c80366d0 msgr2=0x7f17c8038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 --2- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f17c80366d0 0x7f17c8038b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f17d400ad90 tx=0x7f17d4009410 comp rx=0 tx=0).stop 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 msgr2=0x7f17e00f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 --2- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 0x7f17e00f8f80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17d800b690 tx=0x7f17d8009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 shutdown_connections 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 --2- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f17c80366d0 0x7f17c8038b80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 --2- 172.21.15.89:0/1077650454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f17e00fb760 0x7f17e00f8f80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 >> 172.21.15.89:0/1077650454 conn(0x7f17e00f6ec0 msgr2=0x7f17e0066040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 shutdown_connections 2023-12-19T18:40:56.612 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:56.612+0000 7f17e5cc3700 1 -- 172.21.15.89:0/1077650454 wait complete. 2023-12-19T18:40:56.613 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:40:57.197 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:56 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/1077650454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:40:58.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:57 smithi059 ceph-mon[80661]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:40:58.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:40:58.223 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:40:58.469 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:58.514 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:40:59.926 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.926+0000 7f4cc1e69700 1 -- 172.21.15.89:0/683019650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 msgr2=0x7f4cbc0fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:40:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.926+0000 7f4cc1e69700 1 --2- 172.21.15.89:0/683019650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 0x7f4cbc0fbb80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4cac0048c0 tx=0x7f4cac009e30 comp rx=0 tx=0).stop 2023-12-19T18:40:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.926+0000 7f4cc1e69700 1 -- 172.21.15.89:0/683019650 shutdown_connections 2023-12-19T18:40:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.926+0000 7f4cc1e69700 1 --2- 172.21.15.89:0/683019650 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 0x7f4cbc0fbb80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:40:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.926+0000 7f4cc1e69700 1 -- 172.21.15.89:0/683019650 >> 172.21.15.89:0/683019650 conn(0x7f4cbc0f6e50 msgr2=0x7f4cbc0f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:40:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.926+0000 7f4cc1e69700 1 -- 172.21.15.89:0/683019650 shutdown_connections 2023-12-19T18:40:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.926+0000 7f4cc1e69700 1 -- 172.21.15.89:0/683019650 wait complete. 2023-12-19T18:40:59.927 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.927+0000 7f4cc1e69700 1 Processor -- start 2023-12-19T18:40:59.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.927+0000 7f4cc1e69700 1 -- start start 2023-12-19T18:40:59.928 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4cc1e69700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 0x7f4cbc10f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4cc1e69700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0664a0 con 0x7f4cbc0fb770 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4cbb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 0x7f4cbc10f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4cbb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 0x7f4cbc10f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:49646/0 (socket says 172.21.15.89:49646) 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4cbb7fe700 1 -- 172.21.15.89:0/317259800 learned_addr learned my addr 172.21.15.89:0/317259800 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4cbb7fe700 1 -- 172.21.15.89:0/317259800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac005040 con 0x7f4cbc0fb770 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4cbb7fe700 1 --2- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 0x7f4cbc10f7e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009330 tx=0x7f4cac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4ca3fff700 1 -- 172.21.15.89:0/317259800 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cac0096b0 con 0x7f4cbc0fb770 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc0666a0 con 0x7f4cbc0fb770 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4ca3fff700 1 -- 172.21.15.89:0/317259800 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cac009810 con 0x7f4cbc0fb770 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4ca3fff700 1 -- 172.21.15.89:0/317259800 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cac01c5b0 con 0x7f4cbc0fb770 2023-12-19T18:40:59.929 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.928+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc10ff60 con 0x7f4cbc0fb770 2023-12-19T18:40:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.929+0000 7f4ca3fff700 1 -- 172.21.15.89:0/317259800 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f4cac01c870 con 0x7f4cbc0fb770 2023-12-19T18:40:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.929+0000 7f4ca3fff700 1 --2- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f4ca40366d0 0x7f4ca4038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:40:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.929+0000 7f4ca3fff700 1 -- 172.21.15.89:0/317259800 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4cac049ce0 con 0x7f4cbc0fb770 2023-12-19T18:40:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.929+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc105400 con 0x7f4cbc0fb770 2023-12-19T18:40:59.930 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.930+0000 7f4cbaffd700 1 --2- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f4ca40366d0 0x7f4ca4038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:40:59.931 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.930+0000 7f4cbaffd700 1 --2- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f4ca40366d0 0x7f4ca4038b80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009990 tx=0x7f4cb0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:40:59.934 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:40:59.935+0000 7f4ca3fff700 1 -- 172.21.15.89:0/317259800 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4cac011dd0 con 0x7f4cbc0fb770 2023-12-19T18:41:00.191 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.191+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4cbc1013f0 con 0x7f4cbc0fb770 2023-12-19T18:41:00.192 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.192+0000 7f4ca3fff700 1 -- 172.21.15.89:0/317259800 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4cac011370 con 0x7f4cbc0fb770 2023-12-19T18:41:00.192 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:41:00.193 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:41:00.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f4ca40366d0 msgr2=0x7f4ca4038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:00.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 --2- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f4ca40366d0 0x7f4ca4038b80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009990 tx=0x7f4cb0006e40 comp rx=0 tx=0).stop 2023-12-19T18:41:00.194 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 msgr2=0x7f4cbc10f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:00.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 --2- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 0x7f4cbc10f7e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009330 tx=0x7f4cac009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:00.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 shutdown_connections 2023-12-19T18:41:00.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 --2- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f4ca40366d0 0x7f4ca4038b80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:00.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 --2- 172.21.15.89:0/317259800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4cbc0fb770 0x7f4cbc10f7e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:00.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 >> 172.21.15.89:0/317259800 conn(0x7f4cbc0f6e50 msgr2=0x7f4cbc0f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:00.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 shutdown_connections 2023-12-19T18:41:00.195 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:00.195+0000 7f4cc1e69700 1 -- 172.21.15.89:0/317259800 wait complete. 2023-12-19T18:41:00.196 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:40:59 smithi059 ceph-mon[80661]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:00.196 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:41:01.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:00 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/317259800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:01.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:00 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:01.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:00 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:01.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:00 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:01.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:00 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-19T18:41:01.751 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:41:01.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:41:01.994 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:41:02.041 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:41:02.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:01 smithi059 ceph-mon[80661]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:02.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:01 smithi059 ceph-mon[80661]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-19T18:41:02.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:01 smithi059 ceph-mon[80661]: mgrmap e14: smithi059.ccbnob(active, since 62s) 2023-12-19T18:41:03.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.306+0000 7f09334d8700 1 -- 172.21.15.89:0/1048936733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 msgr2=0x7f092c0fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:03.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.306+0000 7f09334d8700 1 --2- 172.21.15.89:0/1048936733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 0x7f092c0fbb80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f09280048c0 tx=0x7f0928009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:03.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.306+0000 7f09334d8700 1 -- 172.21.15.89:0/1048936733 shutdown_connections 2023-12-19T18:41:03.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.306+0000 7f09334d8700 1 --2- 172.21.15.89:0/1048936733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 0x7f092c0fbb80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:03.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.306+0000 7f09334d8700 1 -- 172.21.15.89:0/1048936733 >> 172.21.15.89:0/1048936733 conn(0x7f092c0f6e50 msgr2=0x7f092c0f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:03.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.306+0000 7f09334d8700 1 -- 172.21.15.89:0/1048936733 shutdown_connections 2023-12-19T18:41:03.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.306+0000 7f09334d8700 1 -- 172.21.15.89:0/1048936733 wait complete. 2023-12-19T18:41:03.308 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.307+0000 7f09334d8700 1 Processor -- start 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.307+0000 7f09334d8700 1 -- start start 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.307+0000 7f09334d8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 0x7f092c10f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.307+0000 7f09334d8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092c0664a0 con 0x7f092c0fb770 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.307+0000 7f0931274700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 0x7f092c10f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.307+0000 7f0931274700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 0x7f092c10f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:55694/0 (socket says 172.21.15.89:55694) 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.307+0000 7f0931274700 1 -- 172.21.15.89:0/1646241025 learned_addr learned my addr 172.21.15.89:0/1646241025 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.308+0000 7f0931274700 1 -- 172.21.15.89:0/1646241025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0928005040 con 0x7f092c0fb770 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.308+0000 7f0931274700 1 --2- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 0x7f092c10f7e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0928009330 tx=0x7f0928009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.308+0000 7f0921ffb700 1 -- 172.21.15.89:0/1646241025 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09280096b0 con 0x7f092c0fb770 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.308+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c0666a0 con 0x7f092c0fb770 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.308+0000 7f0921ffb700 1 -- 172.21.15.89:0/1646241025 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0928009810 con 0x7f092c0fb770 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.308+0000 7f0921ffb700 1 -- 172.21.15.89:0/1646241025 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f092801c5b0 con 0x7f092c0fb770 2023-12-19T18:41:03.309 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.308+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092c10ff60 con 0x7f092c0fb770 2023-12-19T18:41:03.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.309+0000 7f0921ffb700 1 -- 172.21.15.89:0/1646241025 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f0928013420 con 0x7f092c0fb770 2023-12-19T18:41:03.310 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.309+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092c105400 con 0x7f092c0fb770 2023-12-19T18:41:03.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.309+0000 7f0921ffb700 1 --2- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0918036720 0x7f0918038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:03.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.309+0000 7f0921ffb700 1 -- 172.21.15.89:0/1646241025 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0928049ae0 con 0x7f092c0fb770 2023-12-19T18:41:03.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.309+0000 7f0930a73700 1 -- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0918036720 msgr2=0x7f0918038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/6145510 2023-12-19T18:41:03.311 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.309+0000 7f0930a73700 1 --2- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0918036720 0x7f0918038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T18:41:03.313 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.314+0000 7f0921ffb700 1 -- 172.21.15.89:0/1646241025 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f09280119c0 con 0x7f092c0fb770 2023-12-19T18:41:03.509 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.510+0000 7f0930a73700 1 -- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0918036720 msgr2=0x7f0918038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/6145510 2023-12-19T18:41:03.510 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.510+0000 7f0930a73700 1 --2- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0918036720 0x7f0918038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T18:41:03.574 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.575+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f092c04bb80 con 0x7f092c0fb770 2023-12-19T18:41:03.575 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.575+0000 7f0921ffb700 1 -- 172.21.15.89:0/1646241025 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0928011370 con 0x7f092c0fb770 2023-12-19T18:41:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:41:03.576 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0918036720 msgr2=0x7f0918038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 --2- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0918036720 0x7f0918038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 msgr2=0x7f092c10f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 --2- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 0x7f092c10f7e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0928009330 tx=0x7f0928009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 shutdown_connections 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 --2- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f0918036720 0x7f0918038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 --2- 172.21.15.89:0/1646241025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f092c0fb770 0x7f092c10f7e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 >> 172.21.15.89:0/1646241025 conn(0x7f092c0f6e50 msgr2=0x7f092c0f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 shutdown_connections 2023-12-19T18:41:03.578 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:03.578+0000 7f09334d8700 1 -- 172.21.15.89:0/1646241025 wait complete. 2023-12-19T18:41:03.579 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:41:03.786 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:03 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/1646241025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:05.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:41:05.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:41:05.375 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:41:05.422 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:41:06.805 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.804+0000 7f8335d2c700 1 -- 172.21.15.89:0/1451723860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 msgr2=0x7f83300fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.804+0000 7f8335d2c700 1 --2- 172.21.15.89:0/1451723860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 0x7f83300fcc90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f83200048c0 tx=0x7f8320009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.804+0000 7f8335d2c700 1 -- 172.21.15.89:0/1451723860 shutdown_connections 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.804+0000 7f8335d2c700 1 --2- 172.21.15.89:0/1451723860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 0x7f83300fcc90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.804+0000 7f8335d2c700 1 -- 172.21.15.89:0/1451723860 >> 172.21.15.89:0/1451723860 conn(0x7f83300f7f20 msgr2=0x7f83300fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.804+0000 7f8335d2c700 1 -- 172.21.15.89:0/1451723860 shutdown_connections 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.804+0000 7f8335d2c700 1 -- 172.21.15.89:0/1451723860 wait complete. 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.805+0000 7f8335d2c700 1 Processor -- start 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.805+0000 7f8335d2c700 1 -- start start 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.805+0000 7f8335d2c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 0x7f833010cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.805+0000 7f8335d2c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833010d050 con 0x7f83300fc880 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.805+0000 7f832f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 0x7f833010cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:06.806 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.805+0000 7f832f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 0x7f833010cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:55720/0 (socket says 172.21.15.89:55720) 2023-12-19T18:41:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.805+0000 7f832f7fe700 1 -- 172.21.15.89:0/2615446087 learned_addr learned my addr 172.21.15.89:0/2615446087 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:41:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.805+0000 7f832f7fe700 1 -- 172.21.15.89:0/2615446087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8320005040 con 0x7f83300fc880 2023-12-19T18:41:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.806+0000 7f832f7fe700 1 --2- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 0x7f833010cb10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8320009330 tx=0x7f8320009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.806+0000 7f832cff9700 1 -- 172.21.15.89:0/2615446087 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83200096b0 con 0x7f83300fc880 2023-12-19T18:41:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.806+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833010d250 con 0x7f83300fc880 2023-12-19T18:41:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.806+0000 7f832cff9700 1 -- 172.21.15.89:0/2615446087 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8320009810 con 0x7f83300fc880 2023-12-19T18:41:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.806+0000 7f832cff9700 1 -- 172.21.15.89:0/2615446087 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832001c5b0 con 0x7f83300fc880 2023-12-19T18:41:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.806+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833010def0 con 0x7f83300fc880 2023-12-19T18:41:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.807+0000 7f832cff9700 1 -- 172.21.15.89:0/2615446087 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f8320013420 con 0x7f83300fc880 2023-12-19T18:41:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.807+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833005f270 con 0x7f83300fc880 2023-12-19T18:41:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.807+0000 7f832cff9700 1 --2- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f831c036720 0x7f831c038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.807+0000 7f832cff9700 1 -- 172.21.15.89:0/2615446087 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8320049a20 con 0x7f83300fc880 2023-12-19T18:41:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.807+0000 7f8326ffd700 1 -- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f831c036720 msgr2=0x7f831c038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/6145510 2023-12-19T18:41:06.808 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.807+0000 7f8326ffd700 1 --2- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f831c036720 0x7f831c038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T18:41:06.811 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:06.812+0000 7f832cff9700 1 -- 172.21.15.89:0/2615446087 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f83200119c0 con 0x7f83300fc880 2023-12-19T18:41:07.007 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.008+0000 7f8326ffd700 1 -- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f831c036720 msgr2=0x7f831c038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/6145510 2023-12-19T18:41:07.008 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.008+0000 7f8326ffd700 1 --2- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f831c036720 0x7f831c038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T18:41:07.074 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.074+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f833010ea40 con 0x7f83300fc880 2023-12-19T18:41:07.075 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.075+0000 7f832cff9700 1 -- 172.21.15.89:0/2615446087 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8320011370 con 0x7f83300fc880 2023-12-19T18:41:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:41:07.075 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f831c036720 msgr2=0x7f831c038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 --2- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f831c036720 0x7f831c038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 msgr2=0x7f833010cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 --2- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 0x7f833010cb10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8320009330 tx=0x7f8320009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 shutdown_connections 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 --2- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:6800/6145510,v1:172.21.15.59:6801/6145510] conn(0x7f831c036720 0x7f831c038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 --2- 172.21.15.89:0/2615446087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83300fc880 0x7f833010cb10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 >> 172.21.15.89:0/2615446087 conn(0x7f83300f7f20 msgr2=0x7f83300f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 shutdown_connections 2023-12-19T18:41:07.077 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:07.077+0000 7f8335d2c700 1 -- 172.21.15.89:0/2615446087 wait complete. 2023-12-19T18:41:07.078 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:41:07.944 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:07 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/2615446087' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:08.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:41:08.653 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:41:08.896 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:41:08.943 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:41:09.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: Active manager daemon smithi059.ccbnob restarted 2023-12-19T18:41:09.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: Activating manager daemon smithi059.ccbnob 2023-12-19T18:41:09.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: osdmap e5: 0 total, 0 up, 0 in 2023-12-19T18:41:09.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: mgrmap e15: smithi059.ccbnob(active, starting, since 0.00348852s) 2023-12-19T18:41:09.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:41:09.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr metadata", "who": "smithi059.ccbnob", "id": "smithi059.ccbnob"}]: dispatch 2023-12-19T18:41:09.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T18:41:09.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T18:41:09.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T18:41:09.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: Manager daemon smithi059.ccbnob is now available 2023-12-19T18:41:09.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:41:09.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:41:09.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:09.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:41:09.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/mirror_snapshot_schedule"}]: dispatch 2023-12-19T18:41:09.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/trash_purge_schedule"}]: dispatch 2023-12-19T18:41:10.401 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.401+0000 7f55f5bdc700 1 -- 172.21.15.89:0/819001022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 msgr2=0x7f55f00fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:10.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.401+0000 7f55f5bdc700 1 --2- 172.21.15.89:0/819001022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 0x7f55f00fbb80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f55e00048c0 tx=0x7f55e0009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:10.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.401+0000 7f55f5bdc700 1 -- 172.21.15.89:0/819001022 shutdown_connections 2023-12-19T18:41:10.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.401+0000 7f55f5bdc700 1 --2- 172.21.15.89:0/819001022 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 0x7f55f00fbb80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:10.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.401+0000 7f55f5bdc700 1 -- 172.21.15.89:0/819001022 >> 172.21.15.89:0/819001022 conn(0x7f55f00f6e50 msgr2=0x7f55f00f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:10.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.401+0000 7f55f5bdc700 1 -- 172.21.15.89:0/819001022 shutdown_connections 2023-12-19T18:41:10.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.401+0000 7f55f5bdc700 1 -- 172.21.15.89:0/819001022 wait complete. 2023-12-19T18:41:10.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.402+0000 7f55f5bdc700 1 Processor -- start 2023-12-19T18:41:10.402 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.402+0000 7f55f5bdc700 1 -- start start 2023-12-19T18:41:10.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.402+0000 7f55f5bdc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 0x7f55f0111b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:10.403 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.402+0000 7f55f5bdc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f010d530 con 0x7f55f00fb770 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.402+0000 7f55ef7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 0x7f55f0111b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.402+0000 7f55ef7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 0x7f55f0111b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:55724/0 (socket says 172.21.15.89:55724) 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.402+0000 7f55ef7fe700 1 -- 172.21.15.89:0/3281133801 learned_addr learned my addr 172.21.15.89:0/3281133801 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.403+0000 7f55ef7fe700 1 -- 172.21.15.89:0/3281133801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0005040 con 0x7f55f00fb770 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.403+0000 7f55ef7fe700 1 --2- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 0x7f55f0111b40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009330 tx=0x7f55e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.403+0000 7f55d7fff700 1 -- 172.21.15.89:0/3281133801 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e00096b0 con 0x7f55f00fb770 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.403+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f0066300 con 0x7f55f00fb770 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.403+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f0112240 con 0x7f55f00fb770 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.403+0000 7f55d7fff700 1 -- 172.21.15.89:0/3281133801 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55e0009810 con 0x7f55f00fb770 2023-12-19T18:41:10.404 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.403+0000 7f55d7fff700 1 -- 172.21.15.89:0/3281133801 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e001c5b0 con 0x7f55f00fb770 2023-12-19T18:41:10.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.404+0000 7f55d7fff700 1 -- 172.21.15.89:0/3281133801 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 16) v1 ==== 43202+0+0 (secure 0 0 0) 0x7f55e001c870 con 0x7f55f00fb770 2023-12-19T18:41:10.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.404+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc0052e0 con 0x7f55f00fb770 2023-12-19T18:41:10.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.404+0000 7f55d7fff700 1 --2- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f55d8036560 0x7f55d8038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:10.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.404+0000 7f55d7fff700 1 -- 172.21.15.89:0/3281133801 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f55e00498b0 con 0x7f55f00fb770 2023-12-19T18:41:10.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.405+0000 7f55eeffd700 1 --2- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f55d8036560 0x7f55d8038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:10.405 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.405+0000 7f55eeffd700 1 --2- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f55d8036560 0x7f55d8038a10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009990 tx=0x7f55e4006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:10.409 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.409+0000 7f55d7fff700 1 -- 172.21.15.89:0/3281133801 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f55e0023300 con 0x7f55f00fb770 2023-12-19T18:41:10.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:10 smithi059 ceph-mon[80661]: [19/Dec/2023:18:41:09] ENGINE Bus STARTING 2023-12-19T18:41:10.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:10 smithi059 ceph-mon[80661]: [19/Dec/2023:18:41:09] ENGINE Serving on https://172.21.15.59:7150 2023-12-19T18:41:10.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:10 smithi059 ceph-mon[80661]: [19/Dec/2023:18:41:09] ENGINE Bus STARTED 2023-12-19T18:41:10.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:10 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:10.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:10 smithi059 ceph-mon[80661]: mgrmap e16: smithi059.ccbnob(active, since 1.00586s) 2023-12-19T18:41:10.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:10 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:10.682 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.683+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55dc005590 con 0x7f55f00fb770 2023-12-19T18:41:10.683 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.683+0000 7f55d7fff700 1 -- 172.21.15.89:0/3281133801 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f55e0011b60 con 0x7f55f00fb770 2023-12-19T18:41:10.683 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:41:10.683 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:41:10.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.685+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f55d8036560 msgr2=0x7f55d8038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:10.685 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.685+0000 7f55f5bdc700 1 --2- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f55d8036560 0x7f55d8038a10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009990 tx=0x7f55e4006e40 comp rx=0 tx=0).stop 2023-12-19T18:41:10.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.685+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 msgr2=0x7f55f0111b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:10.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.685+0000 7f55f5bdc700 1 --2- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 0x7f55f0111b40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009330 tx=0x7f55e0009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:10.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.685+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 shutdown_connections 2023-12-19T18:41:10.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.685+0000 7f55f5bdc700 1 --2- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f55d8036560 0x7f55d8038a10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:10.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.685+0000 7f55f5bdc700 1 --2- 172.21.15.89:0/3281133801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f55f00fb770 0x7f55f0111b40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:10.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.685+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 >> 172.21.15.89:0/3281133801 conn(0x7f55f00f6e50 msgr2=0x7f55f00f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:10.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.686+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 shutdown_connections 2023-12-19T18:41:10.686 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:10.686+0000 7f55f5bdc700 1 -- 172.21.15.89:0/3281133801 wait complete. 2023-12-19T18:41:10.686 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:41:11.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:11.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:11 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/3281133801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:11.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:12.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:41:12.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:41:12.943 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:41:12.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:12 smithi059 ceph-mon[80661]: mgrmap e17: smithi059.ccbnob(active, since 3s) 2023-12-19T18:41:13.003 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T18:41:13.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:13 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:13.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:13 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:13.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:13 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:13.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:13 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:13.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:13 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.163+0000 7f35a92e2700 1 -- 172.21.15.89:0/835130698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 msgr2=0x7f35a4102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.163+0000 7f35a92e2700 1 --2- 172.21.15.89:0/835130698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 0x7f35a4102ca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f35980048c0 tx=0x7f3598009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.163+0000 7f35a92e2700 1 -- 172.21.15.89:0/835130698 shutdown_connections 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.163+0000 7f35a92e2700 1 --2- 172.21.15.89:0/835130698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 0x7f35a4102ca0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.163+0000 7f35a92e2700 1 -- 172.21.15.89:0/835130698 >> 172.21.15.89:0/835130698 conn(0x7f35a40f8410 msgr2=0x7f35a40fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.163+0000 7f35a92e2700 1 -- 172.21.15.89:0/835130698 shutdown_connections 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.163+0000 7f35a92e2700 1 -- 172.21.15.89:0/835130698 wait complete. 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.164+0000 7f35a92e2700 1 Processor -- start 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.164+0000 7f35a92e2700 1 -- start start 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.164+0000 7f35a92e2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 0x7f35a4112b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.164+0000 7f35a92e2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a41130a0 con 0x7f35a41028d0 2023-12-19T18:41:15.164 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.164+0000 7f35a2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 0x7f35a4112b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:15.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.164+0000 7f35a2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 0x7f35a4112b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:33494/0 (socket says 172.21.15.89:33494) 2023-12-19T18:41:15.165 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.164+0000 7f35a2ffd700 1 -- 172.21.15.89:0/1687969651 learned_addr learned my addr 172.21.15.89:0/1687969651 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:41:15.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.165+0000 7f35a2ffd700 1 -- 172.21.15.89:0/1687969651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3598005040 con 0x7f35a41028d0 2023-12-19T18:41:15.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.165+0000 7f35a2ffd700 1 --2- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 0x7f35a4112b60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3598009300 tx=0x7f3598009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:15.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.165+0000 7f358b7fe700 1 -- 172.21.15.89:0/1687969651 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3598004e60 con 0x7f35a41028d0 2023-12-19T18:41:15.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.165+0000 7f358b7fe700 1 -- 172.21.15.89:0/1687969651 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3598013420 con 0x7f35a41028d0 2023-12-19T18:41:15.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.165+0000 7f358b7fe700 1 -- 172.21.15.89:0/1687969651 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f359801c5b0 con 0x7f35a41028d0 2023-12-19T18:41:15.166 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.165+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a41132a0 con 0x7f35a41028d0 2023-12-19T18:41:15.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.165+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a410ca20 con 0x7f35a41028d0 2023-12-19T18:41:15.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.166+0000 7f358b7fe700 1 -- 172.21.15.89:0/1687969651 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f359801c7f0 con 0x7f35a41028d0 2023-12-19T18:41:15.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.166+0000 7f358b7fe700 1 --2- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f358c0367c0 0x7f358c038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:15.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.166+0000 7f358b7fe700 1 -- 172.21.15.89:0/1687969651 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f35980499e0 con 0x7f35a41028d0 2023-12-19T18:41:15.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.166+0000 7f35a27fc700 1 --2- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f358c0367c0 0x7f358c038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:15.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.166+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a405f270 con 0x7f35a41028d0 2023-12-19T18:41:15.167 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.166+0000 7f35a27fc700 1 --2- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f358c0367c0 0x7f358c038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3594009990 tx=0x7f3594006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:15.171 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.171+0000 7f358b7fe700 1 -- 172.21.15.89:0/1687969651 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3598023300 con 0x7f35a41028d0 2023-12-19T18:41:15.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:15.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:15.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:15.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:15.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:41:15.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:15.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: Updating smithi059:/etc/ceph/ceph.conf 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: Updating smithi089:/etc/ceph/ceph.conf 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: Updating smithi089:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: Updating smithi059:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T18:41:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-19T18:41:15.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:15.469 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.470+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35a404bc10 con 0x7f35a41028d0 2023-12-19T18:41:15.470 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.470+0000 7f358b7fe700 1 -- 172.21.15.89:0/1687969651 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3598011b60 con 0x7f35a41028d0 2023-12-19T18:41:15.470 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:41:15.470 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:41:15.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f358c0367c0 msgr2=0x7f358c038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:15.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 --2- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f358c0367c0 0x7f358c038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3594009990 tx=0x7f3594006e40 comp rx=0 tx=0).stop 2023-12-19T18:41:15.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 msgr2=0x7f35a4112b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:15.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 --2- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 0x7f35a4112b60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3598009300 tx=0x7f3598009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:15.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 shutdown_connections 2023-12-19T18:41:15.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 --2- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f358c0367c0 0x7f358c038c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:15.472 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 --2- 172.21.15.89:0/1687969651 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35a41028d0 0x7f35a4112b60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:15.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 >> 172.21.15.89:0/1687969651 conn(0x7f35a40f8410 msgr2=0x7f35a40fa3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:15.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 shutdown_connections 2023-12-19T18:41:15.473 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:15.472+0000 7f35a92e2700 1 -- 172.21.15.89:0/1687969651 wait complete. 2023-12-19T18:41:15.473 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:41:16.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:16 smithi059 ceph-mon[80661]: Updating smithi089:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-19T18:41:16.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:16 smithi059 ceph-mon[80661]: Updating smithi059:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-19T18:41:16.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:16 smithi059 ceph-mon[80661]: Deploying daemon crash.smithi089 on smithi089 2023-12-19T18:41:16.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:16 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/1687969651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:17.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:41:17.144 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:41:17.524 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:19.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.376+0000 7f96513ad700 1 -- 172.21.15.89:0/4061538337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 msgr2=0x7f964c0fb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:19.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.376+0000 7f96513ad700 1 --2- 172.21.15.89:0/4061538337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 0x7f964c0fb9b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f963c0048c0 tx=0x7f963c009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:19.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.376+0000 7f96513ad700 1 -- 172.21.15.89:0/4061538337 shutdown_connections 2023-12-19T18:41:19.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.376+0000 7f96513ad700 1 --2- 172.21.15.89:0/4061538337 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 0x7f964c0fb9b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:19.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.376+0000 7f96513ad700 1 -- 172.21.15.89:0/4061538337 >> 172.21.15.89:0/4061538337 conn(0x7f964c0f7310 msgr2=0x7f964c0f9720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:19.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.376+0000 7f96513ad700 1 -- 172.21.15.89:0/4061538337 shutdown_connections 2023-12-19T18:41:19.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.376+0000 7f96513ad700 1 -- 172.21.15.89:0/4061538337 wait complete. 2023-12-19T18:41:19.377 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.376+0000 7f96513ad700 1 Processor -- start 2023-12-19T18:41:19.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.377+0000 7f96513ad700 1 -- start start 2023-12-19T18:41:19.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.377+0000 7f96513ad700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 0x7f964c114cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:19.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.377+0000 7f96513ad700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c115230 con 0x7f964c0fb5e0 2023-12-19T18:41:19.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.377+0000 7f964affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 0x7f964c114cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:19.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.377+0000 7f964affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 0x7f964c114cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:33520/0 (socket says 172.21.15.89:33520) 2023-12-19T18:41:19.378 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.377+0000 7f964affd700 1 -- 172.21.15.89:0/1533181255 learned_addr learned my addr 172.21.15.89:0/1533181255 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:41:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.377+0000 7f964affd700 1 -- 172.21.15.89:0/1533181255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c005040 con 0x7f964c0fb5e0 2023-12-19T18:41:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.378+0000 7f964affd700 1 --2- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 0x7f964c114cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f963c00b880 tx=0x7f963c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.378+0000 7f96337fe700 1 -- 172.21.15.89:0/1533181255 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f963c004e60 con 0x7f964c0fb5e0 2023-12-19T18:41:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.378+0000 7f96337fe700 1 -- 172.21.15.89:0/1533181255 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f963c00bc00 con 0x7f964c0fb5e0 2023-12-19T18:41:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.378+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964c115430 con 0x7f964c0fb5e0 2023-12-19T18:41:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.378+0000 7f96337fe700 1 -- 172.21.15.89:0/1533181255 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f963c01c5b0 con 0x7f964c0fb5e0 2023-12-19T18:41:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.378+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964c115920 con 0x7f964c0fb5e0 2023-12-19T18:41:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.379+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c05f270 con 0x7f964c0fb5e0 2023-12-19T18:41:19.379 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.379+0000 7f96337fe700 1 -- 172.21.15.89:0/1533181255 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f963c01c870 con 0x7f964c0fb5e0 2023-12-19T18:41:19.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.380+0000 7f96337fe700 1 --2- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96340367c0 0x7f9634038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:19.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.380+0000 7f96337fe700 1 -- 172.21.15.89:0/1533181255 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f963c049b80 con 0x7f964c0fb5e0 2023-12-19T18:41:19.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.380+0000 7f964a7fc700 1 --2- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96340367c0 0x7f9634038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:19.380 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.380+0000 7f964a7fc700 1 --2- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96340367c0 0x7f9634038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9640009990 tx=0x7f9640006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:19.385 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.385+0000 7f96337fe700 1 -- 172.21.15.89:0/1533181255 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f963c023300 con 0x7f964c0fb5e0 2023-12-19T18:41:19.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.685+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f964c1103b0 con 0x7f964c0fb5e0 2023-12-19T18:41:19.687 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.686+0000 7f96337fe700 1 -- 172.21.15.89:0/1533181255 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f963c011b60 con 0x7f964c0fb5e0 2023-12-19T18:41:19.687 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:41:19.687 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:41:19.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96340367c0 msgr2=0x7f9634038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:19.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 --2- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96340367c0 0x7f9634038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9640009990 tx=0x7f9640006e40 comp rx=0 tx=0).stop 2023-12-19T18:41:19.688 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 msgr2=0x7f964c114cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:19.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 --2- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 0x7f964c114cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f963c00b880 tx=0x7f963c009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:19.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 shutdown_connections 2023-12-19T18:41:19.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 --2- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96340367c0 0x7f9634038c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:19.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 --2- 172.21.15.89:0/1533181255 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f964c0fb5e0 0x7f964c114cf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:19.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 >> 172.21.15.89:0/1533181255 conn(0x7f964c0f7310 msgr2=0x7f964c0f8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:19.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 shutdown_connections 2023-12-19T18:41:19.689 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:19.688+0000 7f96513ad700 1 -- 172.21.15.89:0/1533181255 wait complete. 2023-12-19T18:41:19.690 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:41:20.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:20.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:20.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:20.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:20.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:20 smithi059 ceph-mon[80661]: Deploying daemon node-exporter.smithi089 on smithi089 2023-12-19T18:41:20.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:20 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/1533181255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:21.330 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:41:21.330 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:41:21.585 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.705+0000 7f662399e700 1 -- 172.21.15.89:0/401947581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 msgr2=0x7f661c0fb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.705+0000 7f662399e700 1 --2- 172.21.15.89:0/401947581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 0x7f661c0fb420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f66180048c0 tx=0x7f6618009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.705+0000 7f662399e700 1 -- 172.21.15.89:0/401947581 shutdown_connections 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.705+0000 7f662399e700 1 --2- 172.21.15.89:0/401947581 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 0x7f661c0fb420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.705+0000 7f662399e700 1 -- 172.21.15.89:0/401947581 >> 172.21.15.89:0/401947581 conn(0x7f661c071d60 msgr2=0x7f661c072160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.705+0000 7f662399e700 1 -- 172.21.15.89:0/401947581 shutdown_connections 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.705+0000 7f662399e700 1 -- 172.21.15.89:0/401947581 wait complete. 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.706+0000 7f662399e700 1 Processor -- start 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.706+0000 7f662399e700 1 -- start start 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.706+0000 7f662399e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 0x7f661c112b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.706+0000 7f662399e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c1130d0 con 0x7f661c0fb050 2023-12-19T18:41:23.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.706+0000 7f662173a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 0x7f661c112b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.706+0000 7f662173a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 0x7f661c112b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:48988/0 (socket says 172.21.15.89:48988) 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.706+0000 7f662173a700 1 -- 172.21.15.89:0/4012669201 learned_addr learned my addr 172.21.15.89:0/4012669201 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.706+0000 7f662173a700 1 -- 172.21.15.89:0/4012669201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6618005040 con 0x7f661c0fb050 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.707+0000 7f662173a700 1 --2- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 0x7f661c112b90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6618009040 tx=0x7f6618009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.707+0000 7f6611ffb700 1 -- 172.21.15.89:0/4012669201 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6618004e60 con 0x7f661c0fb050 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.707+0000 7f6611ffb700 1 -- 172.21.15.89:0/4012669201 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6618009650 con 0x7f661c0fb050 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.707+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c1132d0 con 0x7f661c0fb050 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.707+0000 7f6611ffb700 1 -- 172.21.15.89:0/4012669201 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f661801c3f0 con 0x7f661c0fb050 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.707+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c113740 con 0x7f661c0fb050 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.708+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c05f270 con 0x7f661c0fb050 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.708+0000 7f6611ffb700 1 -- 172.21.15.89:0/4012669201 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f6618013420 con 0x7f661c0fb050 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.708+0000 7f6611ffb700 1 --2- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f66080367c0 0x7f6608038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:23.743 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.708+0000 7f6611ffb700 1 -- 172.21.15.89:0/4012669201 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6618049a50 con 0x7f661c0fb050 2023-12-19T18:41:23.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.708+0000 7f6620f39700 1 --2- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f66080367c0 0x7f6608038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:23.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.709+0000 7f6620f39700 1 --2- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f66080367c0 0x7f6608038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f660c009990 tx=0x7f660c006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:23.744 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.713+0000 7f6611ffb700 1 -- 172.21.15.89:0/4012669201 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6618011550 con 0x7f661c0fb050 2023-12-19T18:41:23.977 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.977+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f661c04bb80 con 0x7f661c0fb050 2023-12-19T18:41:23.978 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.978+0000 7f6611ffb700 1 -- 172.21.15.89:0/4012669201 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6618011370 con 0x7f661c0fb050 2023-12-19T18:41:23.978 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:41:23.978 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:41:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f66080367c0 msgr2=0x7f6608038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 --2- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f66080367c0 0x7f6608038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f660c009990 tx=0x7f660c006e40 comp rx=0 tx=0).stop 2023-12-19T18:41:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 msgr2=0x7f661c112b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 --2- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 0x7f661c112b90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6618009040 tx=0x7f6618009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 shutdown_connections 2023-12-19T18:41:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 --2- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f66080367c0 0x7f6608038c70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 --2- 172.21.15.89:0/4012669201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f661c0fb050 0x7f661c112b90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 >> 172.21.15.89:0/4012669201 conn(0x7f661c071d60 msgr2=0x7f661c100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 shutdown_connections 2023-12-19T18:41:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:23.980+0000 7f662399e700 1 -- 172.21.15.89:0/4012669201 wait complete. 2023-12-19T18:41:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:41:24.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:24.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:41:24.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:24 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/4012669201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:26.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:25 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:26.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:25 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:26.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:25 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:26.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:25 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:26.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:25 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.isaaav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T18:41:26.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:25 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.isaaav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-19T18:41:26.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:25 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T18:41:26.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:25 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:26.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:25 smithi059 ceph-mon[80661]: Deploying daemon mgr.smithi089.isaaav on smithi089 2023-12-19T18:41:26.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:41:26.301 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:41:26.672 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:28.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.446+0000 7f5e7db23700 1 -- 172.21.15.89:0/990143005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 msgr2=0x7f5e78102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:28.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.446+0000 7f5e7db23700 1 --2- 172.21.15.89:0/990143005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 0x7f5e78102ca0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5e680048c0 tx=0x7f5e68009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:28.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.446+0000 7f5e7db23700 1 -- 172.21.15.89:0/990143005 shutdown_connections 2023-12-19T18:41:28.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.446+0000 7f5e7db23700 1 --2- 172.21.15.89:0/990143005 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 0x7f5e78102ca0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:28.448 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.446+0000 7f5e7db23700 1 -- 172.21.15.89:0/990143005 >> 172.21.15.89:0/990143005 conn(0x7f5e780f83f0 msgr2=0x7f5e780fa800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.446+0000 7f5e7db23700 1 -- 172.21.15.89:0/990143005 shutdown_connections 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.446+0000 7f5e7db23700 1 -- 172.21.15.89:0/990143005 wait complete. 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.447+0000 7f5e7db23700 1 Processor -- start 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.447+0000 7f5e7db23700 1 -- start start 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.447+0000 7f5e7db23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 0x7f5e7806ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.447+0000 7f5e7db23700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e78070480 con 0x7f5e781028d0 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e777fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 0x7f5e7806ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e777fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 0x7f5e7806ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.89:49014/0 (socket says 172.21.15.89:49014) 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e777fe700 1 -- 172.21.15.89:0/3867559544 learned_addr learned my addr 172.21.15.89:0/3867559544 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e777fe700 1 -- 172.21.15.89:0/3867559544 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e68005040 con 0x7f5e781028d0 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e777fe700 1 --2- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 0x7f5e7806ff40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009300 tx=0x7f5e68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e5ffff700 1 -- 172.21.15.89:0/3867559544 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e68004e60 con 0x7f5e781028d0 2023-12-19T18:41:28.449 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e78070680 con 0x7f5e781028d0 2023-12-19T18:41:28.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e5ffff700 1 -- 172.21.15.89:0/3867559544 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e68013420 con 0x7f5e781028d0 2023-12-19T18:41:28.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e5ffff700 1 -- 172.21.15.89:0/3867559544 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e6801c5b0 con 0x7f5e781028d0 2023-12-19T18:41:28.450 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.448+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e78070b70 con 0x7f5e781028d0 2023-12-19T18:41:28.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.449+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e78101b40 con 0x7f5e781028d0 2023-12-19T18:41:28.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.450+0000 7f5e5ffff700 1 -- 172.21.15.89:0/3867559544 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f5e6801c7f0 con 0x7f5e781028d0 2023-12-19T18:41:28.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.450+0000 7f5e5ffff700 1 --2- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5e600367c0 0x7f5e60038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:28.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.450+0000 7f5e5ffff700 1 -- 172.21.15.89:0/3867559544 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f5e68049a80 con 0x7f5e781028d0 2023-12-19T18:41:28.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.450+0000 7f5e76ffd700 1 --2- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5e600367c0 0x7f5e60038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:28.451 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.450+0000 7f5e76ffd700 1 --2- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5e600367c0 0x7f5e60038c70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009990 tx=0x7f5e6c006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.455+0000 7f5e5ffff700 1 -- 172.21.15.89:0/3867559544 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5e68023300 con 0x7f5e781028d0 2023-12-19T18:41:28.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.741+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e7804bbb0 con 0x7f5e781028d0 2023-12-19T18:41:28.742 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.742+0000 7f5e5ffff700 1 -- 172.21.15.89:0/3867559544 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5e68011b60 con 0x7f5e781028d0 2023-12-19T18:41:28.742 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:41:28.742 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:38:54.265161Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T18:41:28.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5e600367c0 msgr2=0x7f5e60038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:28.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 --2- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5e600367c0 0x7f5e60038c70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009990 tx=0x7f5e6c006e40 comp rx=0 tx=0).stop 2023-12-19T18:41:28.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 msgr2=0x7f5e7806ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:28.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 --2- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 0x7f5e7806ff40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009300 tx=0x7f5e68009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:28.745 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 shutdown_connections 2023-12-19T18:41:28.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 --2- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5e600367c0 0x7f5e60038c70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:28.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 --2- 172.21.15.89:0/3867559544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e781028d0 0x7f5e7806ff40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:28.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 >> 172.21.15.89:0/3867559544 conn(0x7f5e780f83f0 msgr2=0x7f5e780f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:28.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 shutdown_connections 2023-12-19T18:41:28.746 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:28.745+0000 7f5e7db23700 1 -- 172.21.15.89:0/3867559544 wait complete. 2023-12-19T18:41:28.746 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: Deploying daemon mon.smithi089 on smithi089 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:29.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:29 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/3867559544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:30.757 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T18:41:30.757 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T18:41:31.365 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi089/config 2023-12-19T18:41:31.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:31 smithi059 ceph-mon[80661]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:32.810 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Deploying cephadm binary to smithi089 2023-12-19T18:41:32.811 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Deploying daemon alertmanager.smithi059 on smithi059 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Added host smithi089 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Deploying daemon grafana.smithi059 on smithi059 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/2417141018' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/2417141018' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: osdmap e4: 0 total, 0 up, 0 in 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi089:172.21.15.89=smithi089", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Saving service mon spec with placement smithi059:172.21.15.59=smithi059;smithi089:172.21.15.89=smithi089;count:2 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/1358295199' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/4148485176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/912309668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Deploying daemon prometheus.smithi059 on smithi059 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/693351300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/598160529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/2860594051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.008 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/1077650454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/317259800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14160 172.21.15.59:0/2718720707' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: mgrmap e14: smithi059.ccbnob(active, since 62s) 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/1646241025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/2615446087' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.009 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Active manager daemon smithi059.ccbnob restarted 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Activating manager daemon smithi059.ccbnob 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: osdmap e5: 0 total, 0 up, 0 in 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: mgrmap e15: smithi059.ccbnob(active, starting, since 0.00348852s) 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr metadata", "who": "smithi059.ccbnob", "id": "smithi059.ccbnob"}]: dispatch 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Manager daemon smithi059.ccbnob is now available 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.010 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/mirror_snapshot_schedule"}]: dispatch 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.ccbnob/trash_purge_schedule"}]: dispatch 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: [19/Dec/2023:18:41:09] ENGINE Bus STARTING 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: [19/Dec/2023:18:41:09] ENGINE Serving on https://172.21.15.59:7150 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: [19/Dec/2023:18:41:09] ENGINE Bus STARTED 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: mgrmap e16: smithi059.ccbnob(active, since 1.00586s) 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/3281133801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: mgrmap e17: smithi059.ccbnob(active, since 3s) 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.011 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Updating smithi059:/etc/ceph/ceph.conf 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Updating smithi089:/etc/ceph/ceph.conf 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Updating smithi089:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Updating smithi059:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Updating smithi059:/etc/ceph/ceph.client.admin.keyring 2023-12-19T18:41:33.012 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Updating smithi089:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Updating smithi059:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Deploying daemon crash.smithi089 on smithi089 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/1687969651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.013 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.014 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.014 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.014 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.014 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Deploying daemon node-exporter.smithi089 on smithi089 2023-12-19T18:41:33.014 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/1533181255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.014 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/4012669201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.isaaav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.isaaav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Deploying daemon mgr.smithi089.isaaav on smithi089 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: Deploying daemon mon.smithi089 on smithi089 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/3867559544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: mon.smithi089@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: mon.smithi089@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta expand map: {default=false} 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta from 'false' to 'false' 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta expanded map: {default=false} 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta expand map: {default=info} 2023-12-19T18:41:33.016 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta from 'info' to 'info' 2023-12-19T18:41:33.017 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta expanded map: {default=info} 2023-12-19T18:41:33.017 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta expand map: {default=daemon} 2023-12-19T18:41:33.017 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta from 'daemon' to 'daemon' 2023-12-19T18:41:33.017 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta expanded map: {default=daemon} 2023-12-19T18:41:33.017 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta expand map: {default=debug} 2023-12-19T18:41:33.017 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta from 'debug' to 'debug' 2023-12-19T18:41:33.017 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 ceph-mon[88459]: expand_channel_meta expanded map: {default=debug} 2023-12-19T18:41:33.017 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 bash[88419]: 5ac855c460fa214c6287ae64e685ff4772aee27418fef5fcfef9c2c9e0fcc08d 2023-12-19T18:41:33.017 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:32 smithi089 systemd[1]: Started Ceph mon.smithi089 for bf854274-9e9d-11ee-95a5-87774f69a715. 2023-12-19T18:41:38.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:41:38.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: mon.smithi059 calling monitor election 2023-12-19T18:41:38.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.? 172.21.15.89:0/1013665093' entity='mgr.smithi089.isaaav' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi089.isaaav/crt"}]: dispatch 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: mon.smithi089 calling monitor election 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: mon.smithi059 is new leader, mons smithi059,smithi089 in quorum (ranks 0,1) 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: monmap e2: 2 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],smithi089=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: fsmap 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: osdmap e5: 0 total, 0 up, 0 in 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: mgrmap e17: smithi059.ccbnob(active, since 29s) 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: Standby manager daemon smithi089.isaaav started 2023-12-19T18:41:38.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.? 172.21.15.89:0/1013665093' entity='mgr.smithi089.isaaav' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-19T18:41:38.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: overall HEALTH_OK 2023-12-19T18:41:38.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.? 172.21.15.89:0/1013665093' entity='mgr.smithi089.isaaav' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi089.isaaav/key"}]: dispatch 2023-12-19T18:41:38.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:38.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.? 172.21.15.89:0/1013665093' entity='mgr.smithi089.isaaav' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-19T18:41:38.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:38.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:38.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:38.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:38.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:41:38.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: mon.smithi059 calling monitor election 2023-12-19T18:41:38.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:38.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.? 172.21.15.89:0/1013665093' entity='mgr.smithi089.isaaav' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi089.isaaav/crt"}]: dispatch 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: mon.smithi089 calling monitor election 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: mon.smithi059 is new leader, mons smithi059,smithi089 in quorum (ranks 0,1) 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: monmap e2: 2 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0],smithi089=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: fsmap 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: osdmap e5: 0 total, 0 up, 0 in 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: mgrmap e17: smithi059.ccbnob(active, since 29s) 2023-12-19T18:41:38.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: Standby manager daemon smithi089.isaaav started 2023-12-19T18:41:38.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.? 172.21.15.89:0/1013665093' entity='mgr.smithi089.isaaav' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-19T18:41:38.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: overall HEALTH_OK 2023-12-19T18:41:38.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.? 172.21.15.89:0/1013665093' entity='mgr.smithi089.isaaav' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi089.isaaav/key"}]: dispatch 2023-12-19T18:41:38.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:38.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.? 172.21.15.89:0/1013665093' entity='mgr.smithi089.isaaav' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-19T18:41:38.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:38.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:38.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:38.256 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:39.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:38 smithi059 ceph-mon[80661]: mgrmap e18: smithi059.ccbnob(active, since 29s), standbys: smithi089.isaaav 2023-12-19T18:41:39.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr metadata", "who": "smithi089.isaaav", "id": "smithi089.isaaav"}]: dispatch 2023-12-19T18:41:39.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:41:39.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:38 smithi059 ceph-mon[80661]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:39.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:41:39.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:39.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:39.216 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:38 smithi089 ceph-mon[88459]: mgrmap e18: smithi059.ccbnob(active, since 29s), standbys: smithi089.isaaav 2023-12-19T18:41:39.216 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr metadata", "who": "smithi089.isaaav", "id": "smithi089.isaaav"}]: dispatch 2023-12-19T18:41:39.216 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:41:39.216 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:38 smithi089 ceph-mon[88459]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:39.217 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:41:39.217 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:39.217 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:41:39.355 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.355+0000 7f1f7abfc700 1 -- 172.21.15.89:0/899606619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f640061f0 msgr2=0x7f1f64008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.355+0000 7f1f7abfc700 1 --2- 172.21.15.89:0/899606619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f640061f0 0x7f1f64008650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f740fda50 tx=0x7f1f68002040 comp rx=0 tx=0).stop 2023-12-19T18:41:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.355+0000 7f1f7abfc700 1 -- 172.21.15.89:0/899606619 shutdown_connections 2023-12-19T18:41:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.355+0000 7f1f7abfc700 1 --2- 172.21.15.89:0/899606619 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f640061f0 0x7f1f64008650 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.355+0000 7f1f7abfc700 1 -- 172.21.15.89:0/899606619 >> 172.21.15.89:0/899606619 conn(0x7f1f740f8510 msgr2=0x7f1f740fa920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.355+0000 7f1f7abfc700 1 -- 172.21.15.89:0/899606619 shutdown_connections 2023-12-19T18:41:39.356 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.355+0000 7f1f7abfc700 1 -- 172.21.15.89:0/899606619 wait complete. 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.356+0000 7f1f7abfc700 1 Processor -- start 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.356+0000 7f1f7abfc700 1 -- start start 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f7abfc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f74112de0 0x7f1f741131b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f7abfc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 0x7f1f7410cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f7abfc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7410d360 con 0x7f1f74112de0 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f7abfc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7410d4d0 con 0x7f1f741136f0 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f73fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 0x7f1f7410cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f73fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 0x7f1f7410cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46492/0 (socket says 172.21.15.89:46492) 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f73fff700 1 -- 172.21.15.89:0/806187519 learned_addr learned my addr 172.21.15.89:0/806187519 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f78998700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f74112de0 0x7f1f741131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f73fff700 1 -- 172.21.15.89:0/806187519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 msgr2=0x7f1f7410cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f73fff700 1 -- 172.21.15.89:0/806187519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 msgr2=0x7f1f7410cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-19T18:41:39.358 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f73fff700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 0x7f1f7410cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f73fff700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 0x7f1f7410cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f78998700 1 -- 172.21.15.89:0/806187519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 msgr2=0x7f1f7410cc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f78998700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 0x7f1f7410cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f78998700 1 -- 172.21.15.89:0/806187519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f68005040 con 0x7f1f74112de0 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f78998700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f74112de0 0x7f1f741131b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1f68013000 tx=0x7f1f68002040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f717fa700 1 -- 172.21.15.89:0/806187519 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f68004290 con 0x7f1f74112de0 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f717fa700 1 -- 172.21.15.89:0/806187519 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f68017410 con 0x7f1f74112de0 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.357+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7410d750 con 0x7f1f74112de0 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.358+0000 7f1f717fa700 1 -- 172.21.15.89:0/806187519 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f68022ad0 con 0x7f1f74112de0 2023-12-19T18:41:39.359 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.358+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f74117c00 con 0x7f1f74112de0 2023-12-19T18:41:39.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.358+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7405f270 con 0x7f1f74112de0 2023-12-19T18:41:39.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.359+0000 7f1f717fa700 1 -- 172.21.15.89:0/806187519 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f1f68022c30 con 0x7f1f74112de0 2023-12-19T18:41:39.360 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.359+0000 7f1f717fa700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1f58069020 0x7f1f5806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:39.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.359+0000 7f1f717fa700 1 -- 172.21.15.89:0/806187519 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1f68088040 con 0x7f1f74112de0 2023-12-19T18:41:39.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.360+0000 7f1f73fff700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1f58069020 0x7f1f5806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:39.361 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.360+0000 7f1f73fff700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1f58069020 0x7f1f5806b4d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f60006fd0 tx=0x7f1f60008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:39.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.365+0000 7f1f717fa700 1 -- 172.21.15.89:0/806187519 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1f68059000 con 0x7f1f74112de0 2023-12-19T18:41:39.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.640+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f74104560 con 0x7f1f74112de0 2023-12-19T18:41:39.640 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.641+0000 7f1f717fa700 1 -- 172.21.15.89:0/806187519 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1013 (secure 0 0 0) 0x7f1f680877f0 con 0x7f1f74112de0 2023-12-19T18:41:39.641 INFO:teuthology.orchestra.run.smithi089.stdout: 2023-12-19T18:41:39.641 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":2,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","modified":"2023-12-19T18:41:32.815852Z","created":"2023-12-19T18:38:54.265161Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi089","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-12-19T18:41:39.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1f58069020 msgr2=0x7f1f5806b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:39.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1f58069020 0x7f1f5806b4d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1f60006fd0 tx=0x7f1f60008040 comp rx=0 tx=0).stop 2023-12-19T18:41:39.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f74112de0 msgr2=0x7f1f741131b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:39.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f74112de0 0x7f1f741131b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1f68013000 tx=0x7f1f68002040 comp rx=0 tx=0).stop 2023-12-19T18:41:39.643 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 shutdown_connections 2023-12-19T18:41:39.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1f58069020 0x7f1f5806b4d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:39.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f74112de0 0x7f1f741131b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:39.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 --2- 172.21.15.89:0/806187519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f741136f0 0x7f1f7410cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:39.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 >> 172.21.15.89:0/806187519 conn(0x7f1f740f8510 msgr2=0x7f1f740fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:39.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 shutdown_connections 2023-12-19T18:41:39.644 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:41:39.643+0000 7f1f7abfc700 1 -- 172.21.15.89:0/806187519 wait complete. 2023-12-19T18:41:39.644 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 2 2023-12-19T18:41:40.194 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:39 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/806187519' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:40.195 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-12-19T18:41:40.196 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph config generate-minimal-conf 2023-12-19T18:41:40.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:39 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/806187519' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T18:41:40.476 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:41:41.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:41.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:41:41.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: Updating smithi059:/etc/ceph/ceph.conf 2023-12-19T18:41:41.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: Updating smithi089:/etc/ceph/ceph.conf 2023-12-19T18:41:41.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:41.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: Updating smithi059:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:41.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: Updating smithi089:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:41.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T18:41:41.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-19T18:41:41.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: Updating smithi059:/etc/ceph/ceph.conf 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: Updating smithi089:/etc/ceph/ceph.conf 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: Updating smithi059:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: Updating smithi089:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:41.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T18:41:41.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-19T18:41:41.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:42.566 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:42 smithi059 ceph-mon[80661]: Reconfiguring mon.smithi059 (unknown last config time)... 2023-12-19T18:41:42.566 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:42 smithi059 ceph-mon[80661]: Reconfiguring daemon mon.smithi059 on smithi059 2023-12-19T18:41:42.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.667+0000 7f2cfee44700 1 -- 172.21.15.59:0/4190227089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 msgr2=0x7f2cf80651f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:42.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.667+0000 7f2cfee44700 1 --2- 172.21.15.59:0/4190227089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 0x7f2cf80651f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0048c0 tx=0x7f2cec009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.667+0000 7f2cfee44700 1 -- 172.21.15.59:0/4190227089 shutdown_connections 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.667+0000 7f2cfee44700 1 --2- 172.21.15.59:0/4190227089 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 0x7f2cf80651f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.667+0000 7f2cfee44700 1 -- 172.21.15.59:0/4190227089 >> 172.21.15.59:0/4190227089 conn(0x7f2cf80715e0 msgr2=0x7f2cf80719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.667+0000 7f2cfee44700 1 -- 172.21.15.59:0/4190227089 shutdown_connections 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.667+0000 7f2cfee44700 1 -- 172.21.15.59:0/4190227089 wait complete. 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.668+0000 7f2cfee44700 1 Processor -- start 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.668+0000 7f2cfee44700 1 -- start start 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.668+0000 7f2cfee44700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 0x7f2cf8114e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.668+0000 7f2cfee44700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81153c0 0x7f2cf810ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.668+0000 7f2cfee44700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf81159d0 con 0x7f2cf8064e20 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.668+0000 7f2cfee44700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf810f260 con 0x7f2cf81153c0 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.668+0000 7f2cfcbe0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 0x7f2cf8114e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:42.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.668+0000 7f2cfcbe0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 0x7f2cf8114e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59588/0 (socket says 172.21.15.59:59588) 2023-12-19T18:41:42.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.668+0000 7f2cfcbe0700 1 -- 172.21.15.59:0/1212288916 learned_addr learned my addr 172.21.15.59:0/1212288916 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:41:42.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.669+0000 7f2cfcbe0700 1 -- 172.21.15.59:0/1212288916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81153c0 msgr2=0x7f2cf810ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:42.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.669+0000 7f2cfcbe0700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81153c0 0x7f2cf810ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:42.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.669+0000 7f2cfcbe0700 1 -- 172.21.15.59:0/1212288916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec005040 con 0x7f2cf8064e20 2023-12-19T18:41:42.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.669+0000 7f2cfcbe0700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 0x7f2cf8114e80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00bfe0 tx=0x7f2cec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:42.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.669+0000 7f2cf57fa700 1 -- 172.21.15.59:0/1212288916 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cec00e040 con 0x7f2cf8064e20 2023-12-19T18:41:42.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.669+0000 7f2cf57fa700 1 -- 172.21.15.59:0/1212288916 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2cec013420 con 0x7f2cf8064e20 2023-12-19T18:41:42.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.669+0000 7f2cf57fa700 1 -- 172.21.15.59:0/1212288916 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cec01c630 con 0x7f2cf8064e20 2023-12-19T18:41:42.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.669+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf810f4e0 con 0x7f2cf8064e20 2023-12-19T18:41:42.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.669+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf810faa0 con 0x7f2cf8064e20 2023-12-19T18:41:42.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.670+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf81064b0 con 0x7f2cf8064e20 2023-12-19T18:41:42.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.670+0000 7f2cf57fa700 1 -- 172.21.15.59:0/1212288916 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f2cec0139e0 con 0x7f2cf8064e20 2023-12-19T18:41:42.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.671+0000 7f2cf57fa700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8069040 0x7f2ce806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:42.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.671+0000 7f2cf57fa700 1 -- 172.21.15.59:0/1212288916 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f2cec086f40 con 0x7f2cf8064e20 2023-12-19T18:41:42.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.671+0000 7f2cf7fff700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8069040 0x7f2ce806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:42.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.671+0000 7f2cf7fff700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8069040 0x7f2ce806b4f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4006fd0 tx=0x7f2ce4006dd0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:42.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.679+0000 7f2cf57fa700 1 -- 172.21.15.59:0/1212288916 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2cec058020 con 0x7f2cf8064e20 2023-12-19T18:41:42.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:42 smithi089 ceph-mon[88459]: Reconfiguring mon.smithi059 (unknown last config time)... 2023-12-19T18:41:42.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:42 smithi089 ceph-mon[88459]: Reconfiguring daemon mon.smithi059 on smithi059 2023-12-19T18:41:42.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.878+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2cf804bb80 con 0x7f2cf8064e20 2023-12-19T18:41:42.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.879+0000 7f2cf57fa700 1 -- 172.21.15.59:0/1212288916 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+223 (secure 0 0 0) 0x7f2cec052e60 con 0x7f2cf8064e20 2023-12-19T18:41:42.880 INFO:teuthology.orchestra.run.smithi059.stdout:# minimal ceph.conf for bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:41:42.880 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2023-12-19T18:41:42.880 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = bf854274-9e9d-11ee-95a5-87774f69a715 2023-12-19T18:41:42.880 INFO:teuthology.orchestra.run.smithi059.stdout: mon_host = [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8069040 msgr2=0x7f2ce806b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8069040 0x7f2ce806b4f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4006fd0 tx=0x7f2ce4006dd0 comp rx=0 tx=0).stop 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 msgr2=0x7f2cf8114e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 0x7f2cf8114e80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00bfe0 tx=0x7f2cec009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 shutdown_connections 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8069040 0x7f2ce806b4f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cf8064e20 0x7f2cf8114e80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 --2- 172.21.15.59:0/1212288916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cf81153c0 0x7f2cf810ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 >> 172.21.15.59:0/1212288916 conn(0x7f2cf80715e0 msgr2=0x7f2cf80fa520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 shutdown_connections 2023-12-19T18:41:42.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:42.881+0000 7f2cfee44700 1 -- 172.21.15.59:0/1212288916 wait complete. 2023-12-19T18:41:43.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:43 smithi059 ceph-mon[80661]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:43.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:43 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1212288916' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:43.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:43 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:43.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:43 smithi089 ceph-mon[88459]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:43.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:43 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1212288916' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:43.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:43 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:43.788 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-12-19T18:41:43.788 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:41:43.788 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.conf 2023-12-19T18:41:43.823 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:41:43.824 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-19T18:41:43.895 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:41:43.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.conf 2023-12-19T18:41:43.930 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:41:43.930 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-19T18:41:44.000 INFO:tasks.cephadm:Deploying OSDs... 2023-12-19T18:41:44.000 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:41:44.000 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2023-12-19T18:41:44.017 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-19T18:41:44.017 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2023-12-19T18:41:44.073 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-19T18:41:44.073 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T18:41:44.073 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-12-19T18:41:44.073 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:41:44.073 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T18:41:44.074 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2023-12-19 18:40:59.928518365 +0000 2023-12-19T18:41:44.074 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2023-12-19 18:35:06.800302975 +0000 2023-12-19T18:41:44.074 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2023-12-19 18:35:06.800302975 +0000 2023-12-19T18:41:44.074 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2023-12-19 18:35:06.800302975 +0000 2023-12-19T18:41:44.074 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-19T18:41:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2023-12-19T18:41:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2023-12-19T18:41:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000220486 s, 2.3 MB/s 2023-12-19T18:41:44.144 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-19T18:41:44.202 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2023-12-19T18:41:44.258 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-19T18:41:44.258 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T18:41:44.258 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-12-19T18:41:44.258 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:41:44.258 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T18:41:44.259 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2023-12-19 18:40:59.928518365 +0000 2023-12-19T18:41:44.259 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2023-12-19 18:35:07.334296274 +0000 2023-12-19T18:41:44.259 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2023-12-19 18:35:07.334296274 +0000 2023-12-19T18:41:44.259 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2023-12-19 18:35:07.334296274 +0000 2023-12-19T18:41:44.259 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-19T18:41:44.328 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2023-12-19T18:41:44.329 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2023-12-19T18:41:44.329 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000178967 s, 2.9 MB/s 2023-12-19T18:41:44.330 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-19T18:41:44.386 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2023-12-19T18:41:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-19T18:41:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T18:41:44.441 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-19T18:41:44.441 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:41:44.441 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T18:41:44.441 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2023-12-19 18:40:59.929518352 +0000 2023-12-19T18:41:44.441 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2023-12-19 18:35:07.854289749 +0000 2023-12-19T18:41:44.441 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2023-12-19 18:35:07.854289749 +0000 2023-12-19T18:41:44.441 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2023-12-19 18:35:07.854289749 +0000 2023-12-19T18:41:44.441 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-19T18:41:44.510 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2023-12-19T18:41:44.546 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2023-12-19T18:41:44.546 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000236008 s, 2.2 MB/s 2023-12-19T18:41:44.547 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-19T18:41:44.604 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2023-12-19T18:41:44.659 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-19T18:41:44.659 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T18:41:44.659 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-19T18:41:44.659 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:41:44.659 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T18:41:44.659 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2023-12-19 18:40:59.929518352 +0000 2023-12-19T18:41:44.659 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2023-12-19 18:35:08.380283149 +0000 2023-12-19T18:41:44.659 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2023-12-19 18:35:08.380283149 +0000 2023-12-19T18:41:44.659 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2023-12-19 18:35:08.380283149 +0000 2023-12-19T18:41:44.660 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-19T18:41:44.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:44.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:44 smithi059 ceph-mon[80661]: Reconfiguring mgr.smithi059.ccbnob (unknown last config time)... 2023-12-19T18:41:44.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi059.ccbnob", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T18:41:44.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T18:41:44.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:44.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:44 smithi059 ceph-mon[80661]: Reconfiguring daemon mgr.smithi059.ccbnob on smithi059 2023-12-19T18:41:44.727 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2023-12-19T18:41:44.727 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2023-12-19T18:41:44.727 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000290031 s, 1.8 MB/s 2023-12-19T18:41:44.728 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-19T18:41:44.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:44.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:44 smithi089 ceph-mon[88459]: Reconfiguring mgr.smithi059.ccbnob (unknown last config time)... 2023-12-19T18:41:44.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi059.ccbnob", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T18:41:44.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T18:41:44.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:44.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:44 smithi089 ceph-mon[88459]: Reconfiguring daemon mgr.smithi059.ccbnob on smithi059 2023-12-19T18:41:44.785 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:41:44.785 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2023-12-19T18:41:44.804 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-19T18:41:44.804 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2023-12-19T18:41:44.861 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-19T18:41:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T18:41:44.862 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-19T18:41:44.862 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:41:44.862 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T18:41:44.862 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-12-19 18:41:31.610667013 +0000 2023-12-19T18:41:44.862 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-12-19 18:35:01.366691670 +0000 2023-12-19T18:41:44.862 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-12-19 18:35:01.366691670 +0000 2023-12-19T18:41:44.862 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-12-19 18:35:01.366691670 +0000 2023-12-19T18:41:44.862 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-19T18:41:44.931 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-12-19T18:41:44.932 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-12-19T18:41:44.932 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000212174 s, 2.4 MB/s 2023-12-19T18:41:44.933 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-19T18:41:44.991 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2023-12-19T18:41:45.048 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-19T18:41:45.048 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T18:41:45.048 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-12-19T18:41:45.048 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:41:45.048 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T18:41:45.048 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-12-19 18:41:31.610667013 +0000 2023-12-19T18:41:45.049 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-12-19 18:35:01.868680404 +0000 2023-12-19T18:41:45.049 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-12-19 18:35:01.868680404 +0000 2023-12-19T18:41:45.049 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-12-19 18:35:01.868680404 +0000 2023-12-19T18:41:45.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-19T18:41:45.117 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-12-19T18:41:45.117 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-12-19T18:41:45.117 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.00031571 s, 1.6 MB/s 2023-12-19T18:41:45.118 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-19T18:41:45.176 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2023-12-19T18:41:45.233 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-19T18:41:45.233 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T18:41:45.233 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-19T18:41:45.234 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:41:45.234 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T18:41:45.234 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-12-19 18:41:31.611666990 +0000 2023-12-19T18:41:45.234 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-12-19 18:35:02.364669272 +0000 2023-12-19T18:41:45.234 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-12-19 18:35:02.364669272 +0000 2023-12-19T18:41:45.234 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-12-19 18:35:02.364669272 +0000 2023-12-19T18:41:45.234 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-19T18:41:45.302 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-12-19T18:41:45.302 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-12-19T18:41:45.302 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000303058 s, 1.7 MB/s 2023-12-19T18:41:45.303 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-19T18:41:45.360 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2023-12-19T18:41:45.415 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-19T18:41:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T18:41:45.416 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-19T18:41:45.416 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:41:45.416 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T18:41:45.416 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2023-12-19 18:41:31.611666990 +0000 2023-12-19T18:41:45.416 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-12-19 18:35:02.877657759 +0000 2023-12-19T18:41:45.416 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-12-19 18:35:02.877657759 +0000 2023-12-19T18:41:45.416 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-12-19 18:35:02.877657759 +0000 2023-12-19T18:41:45.416 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-19T18:41:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2023-12-19T18:41:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2023-12-19T18:41:45.483 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.00020492 s, 2.5 MB/s 2023-12-19T18:41:45.484 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-19T18:41:45.532 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:45 smithi059 ceph-mon[80661]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:45.532 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:45.532 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:45.532 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T18:41:45.532 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:45.541 INFO:tasks.cephadm:Deploying osd.0 on smithi059 with /dev/vg_nvme/lv_4... 2023-12-19T18:41:45.541 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-19T18:41:45.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:45 smithi089 ceph-mon[88459]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:45.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:45.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:45.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T18:41:45.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:46.235 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:41:46.646 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:46 smithi059 ceph-mon[80661]: Reconfiguring crash.smithi059 (monmap changed)... 2023-12-19T18:41:46.646 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:46 smithi059 ceph-mon[80661]: Reconfiguring daemon crash.smithi059 on smithi059 2023-12-19T18:41:46.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:46.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:46.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:46 smithi089 ceph-mon[88459]: Reconfiguring crash.smithi059 (monmap changed)... 2023-12-19T18:41:46.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:46 smithi089 ceph-mon[88459]: Reconfiguring daemon crash.smithi059 on smithi059 2023-12-19T18:41:46.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:46.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:47.581 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:47 smithi059 ceph-mon[80661]: Reconfiguring alertmanager.smithi059 (dependencies changed)... 2023-12-19T18:41:47.581 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:47 smithi059 ceph-mon[80661]: Reconfiguring daemon alertmanager.smithi059 on smithi059 2023-12-19T18:41:47.581 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:47 smithi059 ceph-mon[80661]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:47.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:47 smithi089 ceph-mon[88459]: Reconfiguring alertmanager.smithi059 (dependencies changed)... 2023-12-19T18:41:47.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:47 smithi089 ceph-mon[88459]: Reconfiguring daemon alertmanager.smithi059 on smithi059 2023-12-19T18:41:47.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:47 smithi089 ceph-mon[88459]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:48.754 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-19T18:41:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-19T18:41:48.755 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2023-12-19T18:41:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2023-12-19T18:41:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119533 s, 877 MB/s 2023-12-19T18:41:48.755 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2023-12-19T18:41:49.945 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:49 smithi089 ceph-mon[88459]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:49.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:49 smithi059 ceph-mon[80661]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:50.250 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_4 2023-12-19T18:41:50.761 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:41:51.873 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:51 smithi059 ceph-mon[80661]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:52.003 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:51 smithi089 ceph-mon[88459]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:53.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.109+0000 7fb12e80e700 1 -- 172.21.15.59:0/454834518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 msgr2=0x7fb128102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:53.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.109+0000 7fb12e80e700 1 --2- 172.21.15.59:0/454834518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 0x7fb128102430 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb11c004940 tx=0x7fb11c009e30 comp rx=0 tx=0).stop 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.109+0000 7fb12e80e700 1 -- 172.21.15.59:0/454834518 shutdown_connections 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.109+0000 7fb12e80e700 1 --2- 172.21.15.59:0/454834518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 0x7fb128102430 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.109+0000 7fb12e80e700 1 --2- 172.21.15.59:0/454834518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280fd730 0x7fb1280ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.109+0000 7fb12e80e700 1 -- 172.21.15.59:0/454834518 >> 172.21.15.59:0/454834518 conn(0x7fb1280f7190 msgr2=0x7fb1280f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.109+0000 7fb12e80e700 1 -- 172.21.15.59:0/454834518 shutdown_connections 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.109+0000 7fb12e80e700 1 -- 172.21.15.59:0/454834518 wait complete. 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.109+0000 7fb12e80e700 1 Processor -- start 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb12e80e700 1 -- start start 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb12e80e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280fd730 0x7fb12810d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb12e80e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 0x7fb12810da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb12e80e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12810e070 con 0x7fb128100050 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb12e80e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12810e910 con 0x7fb1280fd730 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb1277fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 0x7fb12810da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb1277fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 0x7fb12810da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:59970/0 (socket says 172.21.15.59:59970) 2023-12-19T18:41:53.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb1277fe700 1 -- 172.21.15.59:0/2084603975 learned_addr learned my addr 172.21.15.59:0/2084603975 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb127fff700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280fd730 0x7fb12810d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb1277fe700 1 -- 172.21.15.59:0/2084603975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280fd730 msgr2=0x7fb12810d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb1277fe700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280fd730 0x7fb12810d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.110+0000 7fb1277fe700 1 -- 172.21.15.59:0/2084603975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c005040 con 0x7fb128100050 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.111+0000 7fb1277fe700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 0x7fb12810da70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009fd0 tx=0x7fb11c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.111+0000 7fb124ff9700 1 -- 172.21.15.59:0/2084603975 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb11c00e040 con 0x7fb128100050 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.111+0000 7fb124ff9700 1 -- 172.21.15.59:0/2084603975 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb11c004290 con 0x7fb128100050 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.111+0000 7fb124ff9700 1 -- 172.21.15.59:0/2084603975 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb11c01c3f0 con 0x7fb128100050 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.111+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12810eb90 con 0x7fb128100050 2023-12-19T18:41:53.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.111+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12810f0d0 con 0x7fb128100050 2023-12-19T18:41:53.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.112+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1281075c0 con 0x7fb128100050 2023-12-19T18:41:53.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.112+0000 7fb124ff9700 1 -- 172.21.15.59:0/2084603975 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7fb11c00bc50 con 0x7fb128100050 2023-12-19T18:41:53.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.113+0000 7fb124ff9700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb110068ed0 0x7fb11006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:41:53.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.113+0000 7fb124ff9700 1 -- 172.21.15.59:0/2084603975 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb11c086f80 con 0x7fb128100050 2023-12-19T18:41:53.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.113+0000 7fb127fff700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb110068ed0 0x7fb11006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:41:53.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.113+0000 7fb127fff700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb110068ed0 0x7fb11006b380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb118004f40 tx=0x7fb11800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:41:53.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.117+0000 7fb124ff9700 1 -- 172.21.15.59:0/2084603975 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb11c0b8a00 con 0x7fb128100050 2023-12-19T18:41:53.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:52 smithi059 ceph-mon[80661]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:53.253 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:52 smithi089 ceph-mon[88459]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:53.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:41:53.339+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb12810f340 con 0x7fb110068ed0 2023-12-19T18:41:53.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:53 smithi059 ceph-mon[80661]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:41:53.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:41:53.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:41:53.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:53.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:41:53.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:53.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:54.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:53 smithi089 ceph-mon[88459]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:41:54.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:41:54.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:41:54.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:41:54.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:41:54.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:54.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:54.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:54 smithi059 ceph-mon[80661]: Reconfiguring grafana.smithi059 (dependencies changed)... 2023-12-19T18:41:54.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:54 smithi059 ceph-mon[80661]: Reconfiguring daemon grafana.smithi059 on smithi059 2023-12-19T18:41:54.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:54 smithi059 ceph-mon[80661]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:55.253 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:54 smithi089 ceph-mon[88459]: Reconfiguring grafana.smithi059 (dependencies changed)... 2023-12-19T18:41:55.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:54 smithi089 ceph-mon[88459]: Reconfiguring daemon grafana.smithi059 on smithi059 2023-12-19T18:41:55.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:54 smithi089 ceph-mon[88459]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:57.647 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:57 smithi059 ceph-mon[80661]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:58.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:57 smithi089 ceph-mon[88459]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:59.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:59 smithi059 ceph-mon[80661]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:41:59.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:41:59.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:41:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:00.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:59 smithi089 ceph-mon[88459]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:00.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:00.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:41:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:00.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:00 smithi059 ceph-mon[80661]: Reconfiguring prometheus.smithi059 (dependencies changed)... 2023-12-19T18:42:00.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:00 smithi059 ceph-mon[80661]: Reconfiguring daemon prometheus.smithi059 on smithi059 2023-12-19T18:42:00.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:00 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1150185427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a33e1310-4169-41e4-bd1f-748898ef9289"}]: dispatch 2023-12-19T18:42:00.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:00 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1150185427' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a33e1310-4169-41e4-bd1f-748898ef9289"}]': finished 2023-12-19T18:42:00.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:00 smithi059 ceph-mon[80661]: osdmap e6: 1 total, 0 up, 1 in 2023-12-19T18:42:00.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:00 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:00.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:00 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2299881775' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:42:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:00 smithi089 ceph-mon[88459]: Reconfiguring prometheus.smithi059 (dependencies changed)... 2023-12-19T18:42:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:00 smithi089 ceph-mon[88459]: Reconfiguring daemon prometheus.smithi059 on smithi059 2023-12-19T18:42:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:00 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1150185427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a33e1310-4169-41e4-bd1f-748898ef9289"}]: dispatch 2023-12-19T18:42:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:00 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1150185427' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a33e1310-4169-41e4-bd1f-748898ef9289"}]': finished 2023-12-19T18:42:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:00 smithi089 ceph-mon[88459]: osdmap e6: 1 total, 0 up, 1 in 2023-12-19T18:42:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:00 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:00 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/2299881775' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:42:01.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:01 smithi059 ceph-mon[80661]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:02.003 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:01 smithi089 ceph-mon[88459]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:03.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:02 smithi059 ceph-mon[80661]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:03.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:02 smithi089 ceph-mon[88459]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:05.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:05 smithi059 ceph-mon[80661]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:06.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:05 smithi089 ceph-mon[88459]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:07.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:07 smithi059 ceph-mon[80661]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:08.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:07 smithi089 ceph-mon[88459]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:08.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:42:08.905 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:08 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:42:09.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:09 smithi059 ceph-mon[80661]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:09.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:09.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:09.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T18:42:09.822 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:09.838 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:09 smithi089 ceph-mon[88459]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:09.838 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:09.839 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:09.839 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi089", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T18:42:09.839 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:10.608 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:10 smithi059 ceph-mon[80661]: Reconfiguring crash.smithi089 (monmap changed)... 2023-12-19T18:42:10.608 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:10 smithi059 ceph-mon[80661]: Reconfiguring daemon crash.smithi089 on smithi089 2023-12-19T18:42:10.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:10 smithi089 ceph-mon[88459]: Reconfiguring crash.smithi089 (monmap changed)... 2023-12-19T18:42:10.978 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:10 smithi089 ceph-mon[88459]: Reconfiguring daemon crash.smithi089 on smithi089 2023-12-19T18:42:11.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:11.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:11.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:11.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-19T18:42:11.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:11.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: Deploying daemon osd.0 on smithi059 2023-12-19T18:42:11.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: Reconfiguring mgr.smithi089.isaaav (monmap changed)... 2023-12-19T18:42:11.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.isaaav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T18:42:11.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T18:42:11.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:11.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:11 smithi059 ceph-mon[80661]: Reconfiguring daemon mgr.smithi089.isaaav on smithi089 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: Deploying daemon osd.0 on smithi059 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: Reconfiguring mgr.smithi089.isaaav (monmap changed)... 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.isaaav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T18:42:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:12.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:11 smithi089 ceph-mon[88459]: Reconfiguring daemon mgr.smithi089.isaaav on smithi089 2023-12-19T18:42:12.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:12 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:12.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:12 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:12.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:12 smithi059 ceph-mon[80661]: Reconfiguring mon.smithi089 (monmap changed)... 2023-12-19T18:42:12.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:12 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T18:42:12.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:12 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-19T18:42:12.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:12 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:12.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:12 smithi059 ceph-mon[80661]: Reconfiguring daemon mon.smithi089 on smithi089 2023-12-19T18:42:12.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:12 smithi059 ceph-mon[80661]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:13.047 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:12 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:13.047 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:12 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:13.047 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:12 smithi089 ceph-mon[88459]: Reconfiguring mon.smithi089 (monmap changed)... 2023-12-19T18:42:13.047 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:12 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T18:42:13.048 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:12 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-19T18:42:13.048 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:12 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:13.048 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:12 smithi089 ceph-mon[88459]: Reconfiguring daemon mon.smithi089 on smithi089 2023-12-19T18:42:13.048 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:12 smithi089 ceph-mon[88459]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:14.418 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.419 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.419 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-19T18:42:14.419 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-19T18:42:14.419 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2023-12-19T18:42:14.419 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2023-12-19T18:42:14.419 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.420 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-19T18:42:14.420 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-19T18:42:14.420 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2023-12-19T18:42:14.420 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2023-12-19T18:42:14.420 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.420 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-19T18:42:14.421 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-19T18:42:14.421 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2023-12-19T18:42:14.421 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2023-12-19T18:42:14.421 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.421 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:14 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:42:14.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi059.front.sepia.ceph.com:9093"}]: dispatch 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi059.front.sepia.ceph.com:3000"}]: dispatch 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-19T18:42:14.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-19T18:42:14.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2023-12-19T18:42:14.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi059.front.sepia.ceph.com:9095"}]: dispatch 2023-12-19T18:42:14.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:14.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:14 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:42:15.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:15 smithi059 ceph-mon[80661]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:15.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:15.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:15 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:15.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:15 smithi089 ceph-mon[88459]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:15.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:15 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:15.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:15 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:17.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:17 smithi059 ceph-mon[80661]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:17.888 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:17 smithi089 ceph-mon[88459]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:19.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:19 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:19.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:19 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:19.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:19 smithi059 ceph-mon[80661]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:19.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:19 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:19.246 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:19 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:19.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:19 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:19.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:19 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:19.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:19 smithi089 ceph-mon[88459]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:19.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:19 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:19.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:19 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:42:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:20.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:42:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:20 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:20 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:42:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:20 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:20 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:42:21.350 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:21 smithi059 ceph-mon[80661]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:21.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:21 smithi089 ceph-mon[88459]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:21.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.666+0000 7fb124ff9700 1 -- 172.21.15.59:0/2084603975 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb12810f340 con 0x7fb110068ed0 2023-12-19T18:42:21.667 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 0 on host 'smithi059' 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb110068ed0 msgr2=0x7fb11006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb110068ed0 0x7fb11006b380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb118004f40 tx=0x7fb11800b040 comp rx=0 tx=0).stop 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 msgr2=0x7fb12810da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 0x7fb12810da70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009fd0 tx=0x7fb11c009e30 comp rx=0 tx=0).stop 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 shutdown_connections 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1280fd730 0x7fb12810d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb110068ed0 0x7fb11006b380 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 --2- 172.21.15.59:0/2084603975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb128100050 0x7fb12810da70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 >> 172.21.15.59:0/2084603975 conn(0x7fb1280f7190 msgr2=0x7fb1280f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:42:21.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 shutdown_connections 2023-12-19T18:42:21.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:21.669+0000 7fb12e80e700 1 -- 172.21.15.59:0/2084603975 wait complete. 2023-12-19T18:42:22.443 DEBUG:teuthology.orchestra.run.smithi059:osd.0> sudo journalctl -f -n 0 -u ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.0.service 2023-12-19T18:42:22.446 INFO:tasks.cephadm:Deploying osd.1 on smithi059 with /dev/vg_nvme/lv_3... 2023-12-19T18:42:22.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-19T18:42:22.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:22 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:22.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:22 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:22.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:22 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:22.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:22 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:22.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:22 smithi059 ceph-mon[80661]: from='osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-19T18:42:22.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:22 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:22.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:22 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:22.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:22 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:22.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:22 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:22.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:22 smithi089 ceph-mon[88459]: from='osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-19T18:42:23.236 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:42:23.621 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:23 smithi059 ceph-mon[80661]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:23.621 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:23 smithi059 ceph-mon[80661]: from='osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-19T18:42:23.621 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:23 smithi059 ceph-mon[80661]: osdmap e7: 1 total, 0 up, 1 in 2023-12-19T18:42:23.621 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:23.621 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:23 smithi059 ceph-mon[80661]: from='osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2023-12-19T18:42:23.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:23 smithi089 ceph-mon[88459]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:23.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:23 smithi089 ceph-mon[88459]: from='osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-19T18:42:23.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:23 smithi089 ceph-mon[88459]: osdmap e7: 1 total, 0 up, 1 in 2023-12-19T18:42:23.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:23.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:23 smithi089 ceph-mon[88459]: from='osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2023-12-19T18:42:24.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:42:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:42:23.900+0000 7f4b1f362700 -1 osd.0 0 waiting for initial osdmap 2023-12-19T18:42:24.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:42:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:42:23.904+0000 7f4b18159700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T18:42:24.628 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:42:24.629 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:24 smithi059 ceph-mon[80661]: from='osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2023-12-19T18:42:24.629 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:24 smithi059 ceph-mon[80661]: osdmap e8: 1 total, 0 up, 1 in 2023-12-19T18:42:24.629 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:24.629 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:42:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:24 smithi089 ceph-mon[88459]: from='osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2023-12-19T18:42:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:24 smithi089 ceph-mon[88459]: osdmap e8: 1 total, 0 up, 1 in 2023-12-19T18:42:24.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:24.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:25.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:25 smithi059 ceph-mon[80661]: purged_snaps scrub starts 2023-12-19T18:42:25.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:25 smithi059 ceph-mon[80661]: purged_snaps scrub ok 2023-12-19T18:42:25.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:25 smithi059 ceph-mon[80661]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:25.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:25 smithi059 ceph-mon[80661]: osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] boot 2023-12-19T18:42:25.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:25 smithi059 ceph-mon[80661]: osdmap e9: 1 total, 1 up, 1 in 2023-12-19T18:42:25.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:25 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:25.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:25 smithi089 ceph-mon[88459]: purged_snaps scrub starts 2023-12-19T18:42:25.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:25 smithi089 ceph-mon[88459]: purged_snaps scrub ok 2023-12-19T18:42:25.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:25 smithi089 ceph-mon[88459]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T18:42:25.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:25 smithi089 ceph-mon[88459]: osd.0 [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] boot 2023-12-19T18:42:25.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:25 smithi089 ceph-mon[88459]: osdmap e9: 1 total, 1 up, 1 in 2023-12-19T18:42:25.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:25 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T18:42:25.893 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-19T18:42:25.893 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-19T18:42:25.893 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2023-12-19T18:42:25.893 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2023-12-19T18:42:25.893 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131584 s, 797 MB/s 2023-12-19T18:42:25.893 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2023-12-19T18:42:26.541 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_3 2023-12-19T18:42:27.074 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:26 smithi059 ceph-mon[80661]: osdmap e10: 1 total, 1 up, 1 in 2023-12-19T18:42:27.075 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:26 smithi059 ceph-mon[80661]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:27.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:26 smithi089 ceph-mon[88459]: osdmap e10: 1 total, 1 up, 1 in 2023-12-19T18:42:27.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:26 smithi089 ceph-mon[88459]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:27.409 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:42:28.474 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:28 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:28.474 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:28 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:28.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:28 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:28.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:28 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:29.538 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:29 smithi059 ceph-mon[80661]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:29.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.545+0000 7f7cbe520700 1 -- 172.21.15.59:0/3313286146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 msgr2=0x7f7cb80fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:42:29.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.545+0000 7f7cbe520700 1 --2- 172.21.15.59:0/3313286146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 0x7f7cb80fd000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8004940 tx=0x7f7ca8009e30 comp rx=0 tx=0).stop 2023-12-19T18:42:29.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.545+0000 7f7cbe520700 1 -- 172.21.15.59:0/3313286146 shutdown_connections 2023-12-19T18:42:29.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.545+0000 7f7cbe520700 1 --2- 172.21.15.59:0/3313286146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cb80fde00 0x7f7cb80fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:29.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.545+0000 7f7cbe520700 1 --2- 172.21.15.59:0/3313286146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 0x7f7cb80fd000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:29.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.545+0000 7f7cbe520700 1 -- 172.21.15.59:0/3313286146 >> 172.21.15.59:0/3313286146 conn(0x7f7cb80f82d0 msgr2=0x7f7cb80fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:42:29.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.545+0000 7f7cbe520700 1 -- 172.21.15.59:0/3313286146 shutdown_connections 2023-12-19T18:42:29.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.545+0000 7f7cbe520700 1 -- 172.21.15.59:0/3313286146 wait complete. 2023-12-19T18:42:29.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.545+0000 7f7cbe520700 1 Processor -- start 2023-12-19T18:42:29.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cbe520700 1 -- start start 2023-12-19T18:42:29.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cbe520700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 0x7f7cb810d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:42:29.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cbe520700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cb80fde00 0x7f7cb810dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:42:29.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cbe520700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb810e170 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cbe520700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb810e9b0 con 0x7f7cb80fde00 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cb7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 0x7f7cb810d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cb7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 0x7f7cb810d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34056/0 (socket says 172.21.15.59:34056) 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cb7fff700 1 -- 172.21.15.59:0/788612522 learned_addr learned my addr 172.21.15.59:0/788612522 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cb77fe700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cb80fde00 0x7f7cb810dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cb7fff700 1 -- 172.21.15.59:0/788612522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cb80fde00 msgr2=0x7f7cb810dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cb7fff700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cb80fde00 0x7f7cb810dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cb7fff700 1 -- 172.21.15.59:0/788612522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca8005040 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.546+0000 7f7cb7fff700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 0x7f7cb810d5a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009fd0 tx=0x7f7ca8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.547+0000 7f7cb4ff9700 1 -- 172.21.15.59:0/788612522 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ca800e040 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.547+0000 7f7cb4ff9700 1 -- 172.21.15.59:0/788612522 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ca800bc50 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.547+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb810ec30 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.547+0000 7f7cb4ff9700 1 -- 172.21.15.59:0/788612522 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ca801c630 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.547+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb810f170 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.548+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb804bb80 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.548+0000 7f7cb4ff9700 1 -- 172.21.15.59:0/788612522 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f7ca8013420 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.548+0000 7f7cb4ff9700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7ca0068ed0 0x7f7ca006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:42:29.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.549+0000 7f7cb4ff9700 1 -- 172.21.15.59:0/788612522 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f7ca80871b0 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.549+0000 7f7cb77fe700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7ca0068ed0 0x7f7ca006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:42:29.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.549+0000 7f7cb77fe700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7ca0068ed0 0x7f7ca006b380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7cac006fd0 tx=0x7f7cac008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:42:29.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.553+0000 7f7cb4ff9700 1 -- 172.21.15.59:0/788612522 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7ca8058030 con 0x7f7cb80fcbf0 2023-12-19T18:42:29.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:29 smithi089 ceph-mon[88459]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:29.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:29.759+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cb810f610 con 0x7f7ca0068ed0 2023-12-19T18:42:30.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:30 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:42:30.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:30 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:42:30.637 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:30 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:30.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:30 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:42:30.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:30 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:42:30.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:30 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:31.733 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:31 smithi059 ceph-mon[80661]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:42:31.733 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:31 smithi059 ceph-mon[80661]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:31.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:31 smithi089 ceph-mon[88459]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:42:31.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:31 smithi089 ceph-mon[88459]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:33.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: Detected new or changed devices on smithi059 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: Adjusting osd_memory_target on smithi059 to 12869M 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:42:33.097 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:32 smithi059 ceph-mon[80661]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: Detected new or changed devices on smithi059 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: Adjusting osd_memory_target on smithi059 to 12869M 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:33.209 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:42:33.210 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:32 smithi089 ceph-mon[88459]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:34.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:33 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2456713337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73a2cb1c-90fe-4d56-adfd-78a688c6b2bf"}]: dispatch 2023-12-19T18:42:34.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:33 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2456713337' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73a2cb1c-90fe-4d56-adfd-78a688c6b2bf"}]': finished 2023-12-19T18:42:34.091 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:33 smithi059 ceph-mon[80661]: osdmap e11: 2 total, 1 up, 2 in 2023-12-19T18:42:34.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:33 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:34.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:33 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:34.092 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:33 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:34.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:33 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/2456713337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73a2cb1c-90fe-4d56-adfd-78a688c6b2bf"}]: dispatch 2023-12-19T18:42:34.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:33 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/2456713337' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73a2cb1c-90fe-4d56-adfd-78a688c6b2bf"}]': finished 2023-12-19T18:42:34.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:33 smithi089 ceph-mon[88459]: osdmap e11: 2 total, 1 up, 2 in 2023-12-19T18:42:34.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:33 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:34.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:33 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:34.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:33 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:35.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:34 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/607649883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:42:35.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:34 smithi059 ceph-mon[80661]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:35.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:34 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/607649883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:42:35.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:34 smithi089 ceph-mon[88459]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:37.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:37.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:37.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:37.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:42:37.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:37 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:37.639 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:37 smithi059 ceph-mon[80661]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:37.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:37.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:37.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:37.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:42:37.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:37 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:37.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:37 smithi089 ceph-mon[88459]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:38.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:42:39.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:42:39.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:39 smithi089 ceph-mon[88459]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:39.901 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:39 smithi059 ceph-mon[80661]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:40.900 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:40 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-19T18:42:40.900 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:40 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:41.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:40 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-19T18:42:41.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:40 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:41.915 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:41 smithi059 ceph-mon[80661]: Deploying daemon osd.1 on smithi059 2023-12-19T18:42:41.915 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:41 smithi059 ceph-mon[80661]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:42.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:41 smithi089 ceph-mon[88459]: Deploying daemon osd.1 on smithi059 2023-12-19T18:42:42.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:41 smithi089 ceph-mon[88459]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:43.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:42 smithi059 ceph-mon[80661]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:43.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:42 smithi089 ceph-mon[88459]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:45.751 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:45 smithi059 ceph-mon[80661]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:46.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:45 smithi089 ceph-mon[88459]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:47.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:47.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:47.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:42:47.133 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:46 smithi059 ceph-mon[80661]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:47.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:47.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:47.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:42:47.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:46 smithi089 ceph-mon[88459]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:48.814 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 1 on host 'smithi059' 2023-12-19T18:42:48.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.813+0000 7f7cb4ff9700 1 -- 172.21.15.59:0/788612522 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7cb810f610 con 0x7f7ca0068ed0 2023-12-19T18:42:48.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7ca0068ed0 msgr2=0x7f7ca006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:42:48.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7ca0068ed0 0x7f7ca006b380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7cac006fd0 tx=0x7f7cac008040 comp rx=0 tx=0).stop 2023-12-19T18:42:48.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 msgr2=0x7f7cb810d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:42:48.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 0x7f7cb810d5a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009fd0 tx=0x7f7ca8009e30 comp rx=0 tx=0).stop 2023-12-19T18:42:48.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 shutdown_connections 2023-12-19T18:42:48.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7ca0068ed0 0x7f7ca006b380 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:48.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7cb80fcbf0 0x7f7cb810d5a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:48.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 --2- 172.21.15.59:0/788612522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7cb80fde00 0x7f7cb810dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:48.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 >> 172.21.15.59:0/788612522 conn(0x7f7cb80f82d0 msgr2=0x7f7cb8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:42:48.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 shutdown_connections 2023-12-19T18:42:48.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:48.816+0000 7f7cbe520700 1 -- 172.21.15.59:0/788612522 wait complete. 2023-12-19T18:42:49.580 DEBUG:teuthology.orchestra.run.smithi059:osd.1> sudo journalctl -f -n 0 -u ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.1.service 2023-12-19T18:42:49.582 INFO:tasks.cephadm:Deploying osd.2 on smithi059 with /dev/vg_nvme/lv_2... 2023-12-19T18:42:49.582 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-19T18:42:49.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:49 smithi089 ceph-mon[88459]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:49.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:49 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:49.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:49 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:49.900 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:49 smithi059 ceph-mon[80661]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:49.900 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:49 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:49.900 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:49 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:49.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:42:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:42:49.686+0000 7f92651a5540 -1 osd.1 0 log_to_monitors true 2023-12-19T18:42:50.357 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:42:50.752 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:50 smithi059 ceph-mon[80661]: from='osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-19T18:42:51.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:50 smithi089 ceph-mon[88459]: from='osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-19T18:42:51.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:51 smithi059 ceph-mon[80661]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:51.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:51 smithi059 ceph-mon[80661]: from='osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-19T18:42:51.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:51 smithi059 ceph-mon[80661]: osdmap e12: 2 total, 1 up, 2 in 2023-12-19T18:42:51.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:51 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:51.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:51 smithi059 ceph-mon[80661]: from='osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2023-12-19T18:42:51.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:42:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:42:51.571+0000 7f925b98a700 -1 osd.1 0 waiting for initial osdmap 2023-12-19T18:42:51.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:42:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:42:51.576+0000 7f9255f84700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T18:42:52.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:51 smithi089 ceph-mon[88459]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:52.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:51 smithi089 ceph-mon[88459]: from='osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-19T18:42:52.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:51 smithi089 ceph-mon[88459]: osdmap e12: 2 total, 1 up, 2 in 2023-12-19T18:42:52.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:51 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:52.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:51 smithi089 ceph-mon[88459]: from='osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2023-12-19T18:42:52.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:52 smithi059 ceph-mon[80661]: from='osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2023-12-19T18:42:52.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:52 smithi059 ceph-mon[80661]: osdmap e13: 2 total, 1 up, 2 in 2023-12-19T18:42:52.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:52 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:52.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:52 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:53.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:52 smithi089 ceph-mon[88459]: from='osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2023-12-19T18:42:53.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:52 smithi089 ceph-mon[88459]: osdmap e13: 2 total, 1 up, 2 in 2023-12-19T18:42:53.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:52 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:53.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:52 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-19T18:42:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-19T18:42:53.445 INFO:teuthology.orchestra.run.smithi059.stderr: stderr: 10+0 records in 2023-12-19T18:42:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2023-12-19T18:42:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124511 s, 842 MB/s 2023-12-19T18:42:53.445 INFO:teuthology.orchestra.run.smithi059.stderr:--> Zapping successful for: 2023-12-19T18:42:53.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:53 smithi059 ceph-mon[80661]: purged_snaps scrub starts 2023-12-19T18:42:53.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:53 smithi059 ceph-mon[80661]: purged_snaps scrub ok 2023-12-19T18:42:53.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:53 smithi059 ceph-mon[80661]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:53.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:53 smithi059 ceph-mon[80661]: osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] boot 2023-12-19T18:42:53.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:53 smithi059 ceph-mon[80661]: osdmap e14: 2 total, 2 up, 2 in 2023-12-19T18:42:53.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:53.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:42:54.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:53 smithi089 ceph-mon[88459]: purged_snaps scrub starts 2023-12-19T18:42:54.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:53 smithi089 ceph-mon[88459]: purged_snaps scrub ok 2023-12-19T18:42:54.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:53 smithi089 ceph-mon[88459]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T18:42:54.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:53 smithi089 ceph-mon[88459]: osd.1 [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] boot 2023-12-19T18:42:54.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:53 smithi089 ceph-mon[88459]: osdmap e14: 2 total, 2 up, 2 in 2023-12-19T18:42:54.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T18:42:54.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:42:54.280 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi059:vg_nvme/lv_2 2023-12-19T18:42:54.559 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:42:54.898 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:54 smithi059 ceph-mon[80661]: osdmap e15: 2 total, 2 up, 2 in 2023-12-19T18:42:54.898 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:54 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:54.899 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:54 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:55.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:54 smithi089 ceph-mon[88459]: osdmap e15: 2 total, 2 up, 2 in 2023-12-19T18:42:55.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:54 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:55.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:54 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:55.812 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:55 smithi059 ceph-mon[80661]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:42:56.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:55 smithi089 ceph-mon[88459]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:42:56.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.215+0000 7faef5fe9700 1 -- 172.21.15.59:0/3093235721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 msgr2=0x7faef00709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:42:56.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.215+0000 7faef5fe9700 1 --2- 172.21.15.59:0/3093235721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 0x7faef00709c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faed8004940 tx=0x7faed8009e30 comp rx=0 tx=0).stop 2023-12-19T18:42:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.215+0000 7faef5fe9700 1 -- 172.21.15.59:0/3093235721 shutdown_connections 2023-12-19T18:42:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.215+0000 7faef5fe9700 1 --2- 172.21.15.59:0/3093235721 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 0x7faef00709c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.215+0000 7faef5fe9700 1 --2- 172.21.15.59:0/3093235721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef0071bc0 0x7faef0070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.215+0000 7faef5fe9700 1 -- 172.21.15.59:0/3093235721 >> 172.21.15.59:0/3093235721 conn(0x7faef00f82d0 msgr2=0x7faef00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:42:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.215+0000 7faef5fe9700 1 -- 172.21.15.59:0/3093235721 shutdown_connections 2023-12-19T18:42:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.215+0000 7faef5fe9700 1 -- 172.21.15.59:0/3093235721 wait complete. 2023-12-19T18:42:56.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.215+0000 7faef5fe9700 1 Processor -- start 2023-12-19T18:42:56.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faef5fe9700 1 -- start start 2023-12-19T18:42:56.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faef5fe9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 0x7faef010f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faef5fe9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef0071bc0 0x7faef010fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faef5fe9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef01103c0 con 0x7faef0070570 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faef5fe9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef0110c60 con 0x7faef0071bc0 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faeef7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 0x7faef010f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faeef7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 0x7faef010f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38554/0 (socket says 172.21.15.59:38554) 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faeef7fe700 1 -- 172.21.15.59:0/1030086781 learned_addr learned my addr 172.21.15.59:0/1030086781 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faeeeffd700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef0071bc0 0x7faef010fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faeef7fe700 1 -- 172.21.15.59:0/1030086781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef0071bc0 msgr2=0x7faef010fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faeef7fe700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef0071bc0 0x7faef010fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faeef7fe700 1 -- 172.21.15.59:0/1030086781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed8005040 con 0x7faef0070570 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.216+0000 7faeef7fe700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 0x7faef010f880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faee0001db0 tx=0x7faee0009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.217+0000 7faee7fff700 1 -- 172.21.15.59:0/1030086781 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faee000ed50 con 0x7faef0070570 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.217+0000 7faee7fff700 1 -- 172.21.15.59:0/1030086781 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faee0007dd0 con 0x7faef0070570 2023-12-19T18:42:56.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.217+0000 7faee7fff700 1 -- 172.21.15.59:0/1030086781 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faee0014640 con 0x7faef0070570 2023-12-19T18:42:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.217+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef0110f40 con 0x7faef0070570 2023-12-19T18:42:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.217+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef0111480 con 0x7faef0070570 2023-12-19T18:42:56.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.218+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef01096f0 con 0x7faef0070570 2023-12-19T18:42:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.218+0000 7faee7fff700 1 -- 172.21.15.59:0/1030086781 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7faee0007920 con 0x7faef0070570 2023-12-19T18:42:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.219+0000 7faee7fff700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7faedc068ed0 0x7faedc06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:42:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.219+0000 7faee7fff700 1 -- 172.21.15.59:0/1030086781 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7faee0085de0 con 0x7faef0070570 2023-12-19T18:42:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.219+0000 7faeeeffd700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7faedc068ed0 0x7faedc06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:42:56.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.219+0000 7faeeeffd700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7faedc068ed0 0x7faedc06b380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faed8004560 tx=0x7faed8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:42:56.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.223+0000 7faee7fff700 1 -- 172.21.15.59:0/1030086781 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7faee004da00 con 0x7faef0070570 2023-12-19T18:42:56.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:42:56.429+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7faef0111750 con 0x7faedc068ed0 2023-12-19T18:42:56.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:56 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:42:56.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:56 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:42:56.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:56 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:57.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:56 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:42:57.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:56 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:42:57.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:56 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:57.823 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:57 smithi059 ceph-mon[80661]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:42:57.823 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:57 smithi059 ceph-mon[80661]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:42:57.886 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:57 smithi089 ceph-mon[88459]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi059:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:42:57.886 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:57 smithi089 ceph-mon[88459]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:42:59.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:42:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:42:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:42:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:42:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:42:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:42:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:42:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:00.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:00.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:00.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:00.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:00.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:00.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:00.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:00.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:00.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:00.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:42:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:00.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:00 smithi059 ceph-mon[80661]: Detected new or changed devices on smithi059 2023-12-19T18:43:00.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:00 smithi059 ceph-mon[80661]: Adjusting osd_memory_target on smithi059 to 6434M 2023-12-19T18:43:00.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:00 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3674393454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0003bcc2-1e16-44d8-a031-8e26fa6c7467"}]: dispatch 2023-12-19T18:43:00.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:00 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3674393454' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0003bcc2-1e16-44d8-a031-8e26fa6c7467"}]': finished 2023-12-19T18:43:00.921 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:00 smithi059 ceph-mon[80661]: osdmap e16: 3 total, 2 up, 3 in 2023-12-19T18:43:00.922 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:00 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:00 smithi089 ceph-mon[88459]: Detected new or changed devices on smithi059 2023-12-19T18:43:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:00 smithi089 ceph-mon[88459]: Adjusting osd_memory_target on smithi059 to 6434M 2023-12-19T18:43:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:00 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3674393454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0003bcc2-1e16-44d8-a031-8e26fa6c7467"}]: dispatch 2023-12-19T18:43:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:00 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3674393454' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0003bcc2-1e16-44d8-a031-8e26fa6c7467"}]': finished 2023-12-19T18:43:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:00 smithi089 ceph-mon[88459]: osdmap e16: 3 total, 2 up, 3 in 2023-12-19T18:43:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:00 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:01.935 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:01 smithi059 ceph-mon[80661]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:01.936 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:01 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3679481635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:43:02.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:01 smithi089 ceph-mon[88459]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:02.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:01 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3679481635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:43:03.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:02 smithi059 ceph-mon[80661]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:03.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:02 smithi089 ceph-mon[88459]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:05.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:04 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:05.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:04 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:05.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:04 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:05.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:04 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:05.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:04 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:05.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:04 smithi059 ceph-mon[80661]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:05.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:04 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:05.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:04 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:05.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:04 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:05.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:04 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:05.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:04 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:05.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:04 smithi089 ceph-mon[88459]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:07.852 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:07 smithi059 ceph-mon[80661]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:07.886 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:07 smithi089 ceph-mon[88459]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:08.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-19T18:43:08.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:08.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:43:09.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:08 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-19T18:43:09.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:08 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:09.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:08 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:43:09.819 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:09 smithi059 ceph-mon[80661]: Deploying daemon osd.2 on smithi059 2023-12-19T18:43:09.819 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:09 smithi059 ceph-mon[80661]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:10.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:09 smithi089 ceph-mon[88459]: Deploying daemon osd.2 on smithi059 2023-12-19T18:43:10.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:09 smithi089 ceph-mon[88459]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:11.814 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:11 smithi059 ceph-mon[80661]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:11 smithi089 ceph-mon[88459]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:13.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:12 smithi059 ceph-mon[80661]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:13.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:12 smithi089 ceph-mon[88459]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:14.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:13 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:14.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:13 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:14.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:13 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:14.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:13 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:14.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:13 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:14.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:13 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:14.969 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:14 smithi059 ceph-mon[80661]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:15.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:14 smithi089 ceph-mon[88459]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:16.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.184+0000 7faee7fff700 1 -- 172.21.15.59:0/1030086781 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faef0111750 con 0x7faedc068ed0 2023-12-19T18:43:16.185 INFO:teuthology.orchestra.run.smithi059.stdout:Created osd(s) 2 on host 'smithi059' 2023-12-19T18:43:16.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7faedc068ed0 msgr2=0x7faedc06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:43:16.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7faedc068ed0 0x7faedc06b380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faed8004560 tx=0x7faed8009e30 comp rx=0 tx=0).stop 2023-12-19T18:43:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 msgr2=0x7faef010f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:43:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 0x7faef010f880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faee0001db0 tx=0x7faee0009ee0 comp rx=0 tx=0).stop 2023-12-19T18:43:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 shutdown_connections 2023-12-19T18:43:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7faedc068ed0 0x7faedc06b380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faef0070570 0x7faef010f880 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 --2- 172.21.15.59:0/1030086781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef0071bc0 0x7faef010fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 >> 172.21.15.59:0/1030086781 conn(0x7faef00f82d0 msgr2=0x7faef0103310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:43:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 shutdown_connections 2023-12-19T18:43:16.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:43:16.186+0000 7faef5fe9700 1 -- 172.21.15.59:0/1030086781 wait complete. 2023-12-19T18:43:16.795 DEBUG:teuthology.orchestra.run.smithi059:osd.2> sudo journalctl -f -n 0 -u ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.2.service 2023-12-19T18:43:16.798 INFO:tasks.cephadm:Deploying osd.3 on smithi089 with /dev/vg_nvme/lv_4... 2023-12-19T18:43:16.798 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-19T18:43:17.055 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi089/config 2023-12-19T18:43:17.431 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:17 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:17.431 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:17 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:17.431 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:17 smithi089 ceph-mon[88459]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:17.431 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:17 smithi089 ceph-mon[88459]: from='osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-19T18:43:17.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:17 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:17.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:17 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:17.433 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:17 smithi059 ceph-mon[80661]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:17.434 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:17 smithi059 ceph-mon[80661]: from='osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-19T18:43:18.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:18 smithi059 ceph-mon[80661]: from='osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-19T18:43:18.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:18 smithi059 ceph-mon[80661]: osdmap e17: 3 total, 2 up, 3 in 2023-12-19T18:43:18.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:18 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:18.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:18 smithi059 ceph-mon[80661]: from='osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2023-12-19T18:43:18.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:43:18.182+0000 7f506f241700 -1 osd.2 0 waiting for initial osdmap 2023-12-19T18:43:18.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:43:18.187+0000 7f506a03c700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T18:43:18.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:18 smithi089 ceph-mon[88459]: from='osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-19T18:43:18.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:18 smithi089 ceph-mon[88459]: osdmap e17: 3 total, 2 up, 3 in 2023-12-19T18:43:18.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:18 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:18.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:18 smithi089 ceph-mon[88459]: from='osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]: dispatch 2023-12-19T18:43:18.715 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-19T18:43:18.715 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-19T18:43:18.715 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2023-12-19T18:43:18.715 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2023-12-19T18:43:18.715 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128348 s, 817 MB/s 2023-12-19T18:43:18.715 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2023-12-19T18:43:19.291 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:19 smithi089 ceph-mon[88459]: from='osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2023-12-19T18:43:19.291 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:19 smithi089 ceph-mon[88459]: osdmap e18: 3 total, 2 up, 3 in 2023-12-19T18:43:19.291 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:19 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:19.291 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:19 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:19.291 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:19 smithi089 ceph-mon[88459]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:19.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_4 2023-12-19T18:43:19.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:19 smithi059 ceph-mon[80661]: from='osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi059", "root=default"]}]': finished 2023-12-19T18:43:19.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:19 smithi059 ceph-mon[80661]: osdmap e18: 3 total, 2 up, 3 in 2023-12-19T18:43:19.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:19 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:19.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:19 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:19.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:19 smithi059 ceph-mon[80661]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T18:43:19.577 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi089/config 2023-12-19T18:43:20.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:20 smithi059 ceph-mon[80661]: purged_snaps scrub starts 2023-12-19T18:43:20.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:20 smithi059 ceph-mon[80661]: purged_snaps scrub ok 2023-12-19T18:43:20.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:20 smithi059 ceph-mon[80661]: osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] boot 2023-12-19T18:43:20.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:20 smithi059 ceph-mon[80661]: osdmap e19: 3 total, 3 up, 3 in 2023-12-19T18:43:20.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:20.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:20.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:20 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:20 smithi089 ceph-mon[88459]: purged_snaps scrub starts 2023-12-19T18:43:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:20 smithi089 ceph-mon[88459]: purged_snaps scrub ok 2023-12-19T18:43:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:20 smithi089 ceph-mon[88459]: osd.2 [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] boot 2023-12-19T18:43:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:20 smithi089 ceph-mon[88459]: osdmap e19: 3 total, 3 up, 3 in 2023-12-19T18:43:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:20 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T18:43:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:20 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:20.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:20 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:20.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.985+0000 7f2d02840700 1 -- 172.21.15.89:0/3628857060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 msgr2=0x7f2cfc0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:43:20.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.985+0000 7f2d02840700 1 --2- 172.21.15.89:0/3628857060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 0x7f2cfc0fbf00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2cec004940 tx=0x7f2cec009e30 comp rx=0 tx=0).stop 2023-12-19T18:43:20.987 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.985+0000 7f2d02840700 1 -- 172.21.15.89:0/3628857060 shutdown_connections 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.985+0000 7f2d02840700 1 --2- 172.21.15.89:0/3628857060 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc0fcd00 0x7f2cfc0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.985+0000 7f2d02840700 1 --2- 172.21.15.89:0/3628857060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 0x7f2cfc0fbf00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.985+0000 7f2d02840700 1 -- 172.21.15.89:0/3628857060 >> 172.21.15.89:0/3628857060 conn(0x7f2cfc0f7190 msgr2=0x7f2cfc0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.985+0000 7f2d02840700 1 -- 172.21.15.89:0/3628857060 shutdown_connections 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.985+0000 7f2d02840700 1 -- 172.21.15.89:0/3628857060 wait complete. 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2d02840700 1 Processor -- start 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2d02840700 1 -- start start 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2d02840700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 0x7f2cfc10c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2d02840700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc0fcd00 0x7f2cfc10c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2d02840700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc10d060 con 0x7f2cfc0fcd00 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2d02840700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc10d8a0 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2cfbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 0x7f2cfc10c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2cfbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 0x7f2cfc10c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48322/0 (socket says 172.21.15.89:48322) 2023-12-19T18:43:20.988 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2cfbfff700 1 -- 172.21.15.89:0/2096327251 learned_addr learned my addr 172.21.15.89:0/2096327251 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2cf3fff700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc0fcd00 0x7f2cfc10c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2cfbfff700 1 -- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc0fcd00 msgr2=0x7f2cfc10c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2cfbfff700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc0fcd00 0x7f2cfc10c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2cfbfff700 1 -- 172.21.15.89:0/2096327251 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec005040 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.986+0000 7f2cfbfff700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 0x7f2cfc10c490 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2cec004f90 tx=0x7f2cec009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.987+0000 7f2cf97fa700 1 -- 172.21.15.89:0/2096327251 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cec00e040 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.987+0000 7f2cf97fa700 1 -- 172.21.15.89:0/2096327251 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2cec00bcd0 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.987+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc10db20 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.987+0000 7f2cf97fa700 1 -- 172.21.15.89:0/2096327251 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cec01c630 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.989 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.987+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc10e060 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.988+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc04bb80 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.988+0000 7f2cf97fa700 1 -- 172.21.15.89:0/2096327251 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f2cec013960 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.988+0000 7f2cf97fa700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8068ed0 0x7f2ce806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:43:20.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.988+0000 7f2cf97fa700 1 -- 172.21.15.89:0/2096327251 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2842+0+0 (secure 0 0 0) 0x7f2cec087610 con 0x7f2cfc0fbaf0 2023-12-19T18:43:20.990 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.989+0000 7f2cf3fff700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8068ed0 0x7f2ce806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:43:20.991 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.989+0000 7f2cf3fff700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8068ed0 0x7f2ce806b380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4006fd0 tx=0x7f2ce4008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:43:20.993 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:20.993+0000 7f2cf97fa700 1 -- 172.21.15.89:0/2096327251 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2cec058130 con 0x7f2cfc0fbaf0 2023-12-19T18:43:21.183 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:21.183+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cfc10e4d0 con 0x7f2ce8068ed0 2023-12-19T18:43:21.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:21 smithi089 ceph-mon[88459]: osdmap e20: 3 total, 3 up, 3 in 2023-12-19T18:43:21.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:21 smithi089 ceph-mon[88459]: pgmap v74: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:21.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:21 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-19T18:43:21.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:21 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:43:21.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:21 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:43:21.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:21 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:21.685 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:21 smithi059 ceph-mon[80661]: osdmap e20: 3 total, 3 up, 3 in 2023-12-19T18:43:21.685 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:21 smithi059 ceph-mon[80661]: pgmap v74: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:21.685 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:21 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-19T18:43:21.685 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:21 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:43:21.686 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:21 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:43:21.686 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:21 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:22.432 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:22 smithi089 ceph-mon[88459]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:43:22.432 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:22 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-19T18:43:22.432 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:22 smithi089 ceph-mon[88459]: osdmap e21: 3 total, 3 up, 3 in 2023-12-19T18:43:22.432 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:22 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-19T18:43:22.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:22 smithi059 ceph-mon[80661]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:43:22.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:22 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-19T18:43:22.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:22 smithi059 ceph-mon[80661]: osdmap e21: 3 total, 3 up, 3 in 2023-12-19T18:43:22.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:22 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: osdmap e22: 3 total, 3 up, 3 in 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: Detected new or changed devices on smithi059 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: Adjusting osd_memory_target on smithi059 to 4289M 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:23.626 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:23.626 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.626 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:23.626 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:23.626 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:23.626 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:23.626 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: osdmap e22: 3 total, 3 up, 3 in 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: Detected new or changed devices on smithi059 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: Adjusting osd_memory_target on smithi059 to 4289M 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:23.756 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:23.756 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:23.756 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:23.756 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:23.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:43:23 smithi059 sudo[118986]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T18:43:23.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:43:23 smithi059 sudo[118986]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:23.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:43:23 smithi059 sudo[118986]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:23.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:43:23 smithi059 sudo[118986]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:24.100 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:24.099+0000 7f2cf97fa700 1 -- 172.21.15.89:0/2096327251 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2cec054a70 con 0x7f2cfc0fbaf0 2023-12-19T18:43:24.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:43:24 smithi059 sudo[118989]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T18:43:24.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:43:24 smithi059 sudo[118989]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:24.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:43:24 smithi059 sudo[118989]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:24.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:43:24 smithi059 sudo[118989]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: osdmap e23: 3 total, 3 up, 3 in 2023-12-19T18:43:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/3090830408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08b40954-a1b3-4754-8494-e3a5423406c9"}]: dispatch 2023-12-19T18:43:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08b40954-a1b3-4754-8494-e3a5423406c9"}]: dispatch 2023-12-19T18:43:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08b40954-a1b3-4754-8494-e3a5423406c9"}]': finished 2023-12-19T18:43:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: osdmap e24: 4 total, 3 up, 4 in 2023-12-19T18:43:24.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:24.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:43:24.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/4195707922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:43:24.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: mgrmap e19: smithi059.ccbnob(active, since 2m), standbys: smithi089.isaaav 2023-12-19T18:43:24.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:24.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:43:24 smithi059 sudo[118992]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T18:43:24.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:43:24 smithi059 sudo[118992]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:24.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:43:24 smithi059 sudo[118992]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:24.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:43:24 smithi059 sudo[118992]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: osdmap e23: 3 total, 3 up, 3 in 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/3090830408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08b40954-a1b3-4754-8494-e3a5423406c9"}]: dispatch 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "08b40954-a1b3-4754-8494-e3a5423406c9"}]: dispatch 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "08b40954-a1b3-4754-8494-e3a5423406c9"}]': finished 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: osdmap e24: 4 total, 3 up, 4 in 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/4195707922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: mgrmap e19: smithi059.ccbnob(active, since 2m), standbys: smithi089.isaaav 2023-12-19T18:43:24.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:25.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[118995]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T18:43:25.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[118995]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:25.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[118995]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:25.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[118995]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:25.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:25 smithi089 ceph-mon[88459]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:25.927 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:25 smithi059 ceph-mon[80661]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:25.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[118998]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T18:43:25.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[118998]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:25.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[118998]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:25.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[118998]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:26.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[119001]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T18:43:26.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[119001]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:26.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[119001]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:26.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:43:25 smithi059 sudo[119001]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:26.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:26 smithi059 sudo[119004]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T18:43:26.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:26 smithi059 sudo[119004]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:26.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:26 smithi059 sudo[119004]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:26.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:26 smithi059 sudo[119004]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:27.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:26 smithi059 sudo[119007]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T18:43:27.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:26 smithi059 sudo[119007]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:27.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:26 smithi059 sudo[119007]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:27.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:26 smithi059 sudo[119007]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:27.332 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 sudo[93070]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T18:43:27.332 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 sudo[93070]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:27.332 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 sudo[93070]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:27.332 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 sudo[93070]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:27.673 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 sudo[93155]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T18:43:27.673 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 sudo[93155]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T18:43:27.673 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 sudo[93155]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T18:43:27.674 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 sudo[93155]: pam_unix(sudo:session): session closed for user root 2023-12-19T18:43:27.674 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 ceph-mon[88459]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-19T18:43:27.674 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 ceph-mon[88459]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:27.674 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 ceph-mon[88459]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-19T18:43:27.674 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:43:27.674 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:43:27.674 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:43:27.674 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:43:27.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:27 smithi059 ceph-mon[80661]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-19T18:43:27.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:27 smithi059 ceph-mon[80661]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:27.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:27 smithi059 ceph-mon[80661]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-19T18:43:27.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:43:27.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:43:27.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-19T18:43:27.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mon metadata", "id": "smithi089"}]: dispatch 2023-12-19T18:43:28.660 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:28 smithi089 ceph-mon[88459]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-19T18:43:28.661 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:28 smithi089 ceph-mon[88459]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-19T18:43:28.661 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:28 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:28.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:28 smithi059 ceph-mon[80661]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-19T18:43:28.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:28 smithi059 ceph-mon[80661]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-19T18:43:28.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:28 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:29.541 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:29 smithi089 ceph-mon[88459]: pgmap v82: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:29.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:29 smithi059 ceph-mon[80661]: pgmap v82: 1 pgs: 1 active+clean; 449 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:31.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:31 smithi059 ceph-mon[80661]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2023-12-19T18:43:31.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-19T18:43:31.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:31.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:31 smithi059 ceph-mon[80661]: Deploying daemon osd.3 on smithi089 2023-12-19T18:43:32.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:31 smithi089 ceph-mon[88459]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2023-12-19T18:43:32.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-19T18:43:32.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:32.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:31 smithi089 ceph-mon[88459]: Deploying daemon osd.3 on smithi089 2023-12-19T18:43:33.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:32 smithi059 ceph-mon[80661]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-12-19T18:43:33.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:32 smithi089 ceph-mon[88459]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2023-12-19T18:43:35.819 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:35 smithi089 ceph-mon[88459]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-19T18:43:35.819 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:35 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:35.819 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:35 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:35.819 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:35 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:35.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:35 smithi059 ceph-mon[80661]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-19T18:43:35.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:35 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:35.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:35 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:35.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:35 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:37.692 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:37 smithi089 ceph-mon[88459]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-12-19T18:43:37.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:37 smithi059 ceph-mon[80661]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-12-19T18:43:38.047 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.046+0000 7f2cf97fa700 1 -- 172.21.15.89:0/2096327251 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2cfc10e4d0 con 0x7f2ce8068ed0 2023-12-19T18:43:38.047 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 3 on host 'smithi089' 2023-12-19T18:43:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8068ed0 msgr2=0x7f2ce806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:43:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8068ed0 0x7f2ce806b380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4006fd0 tx=0x7f2ce4008040 comp rx=0 tx=0).stop 2023-12-19T18:43:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 msgr2=0x7f2cfc10c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:43:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 0x7f2cfc10c490 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2cec004f90 tx=0x7f2cec009e30 comp rx=0 tx=0).stop 2023-12-19T18:43:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 shutdown_connections 2023-12-19T18:43:38.050 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cfc0fbaf0 0x7f2cfc10c490 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:38.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2ce8068ed0 0x7f2ce806b380 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:38.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 --2- 172.21.15.89:0/2096327251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc0fcd00 0x7f2cfc10c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:38.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 >> 172.21.15.89:0/2096327251 conn(0x7f2cfc0f7190 msgr2=0x7f2cfc0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:43:38.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 shutdown_connections 2023-12-19T18:43:38.051 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:38.049+0000 7f2d02840700 1 -- 172.21.15.89:0/2096327251 wait complete. 2023-12-19T18:43:38.797 DEBUG:teuthology.orchestra.run.smithi089:osd.3> sudo journalctl -f -n 0 -u ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.3.service 2023-12-19T18:43:38.800 INFO:tasks.cephadm:Deploying osd.4 on smithi089 with /dev/vg_nvme/lv_3... 2023-12-19T18:43:38.800 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-19T18:43:39.044 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:39 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:39.044 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:39 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:39.044 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:39 smithi089 ceph-mon[88459]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-12-19T18:43:39.044 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:39 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:43:39.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:39 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:39.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:39 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:39.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:39 smithi059 ceph-mon[80661]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-12-19T18:43:39.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:39 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:43:39.504 INFO:journalctl@ceph.osd.3.smithi089.stdout:Dec 19 18:43:39 smithi089 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-3[93903]: 2023-12-19T18:43:39.190+0000 7f62af5ae540 -1 osd.3 0 log_to_monitors true 2023-12-19T18:43:39.539 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi089/config 2023-12-19T18:43:40.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:40 smithi089 ceph-mon[88459]: from='osd.3 [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-19T18:43:40.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:40 smithi089 ceph-mon[88459]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-19T18:43:40.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:40 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:40.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:40 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:40.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:40 smithi059 ceph-mon[80661]: from='osd.3 [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-19T18:43:40.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:40 smithi059 ceph-mon[80661]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-19T18:43:40.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:40 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:40.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:40 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:41.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:41 smithi059 ceph-mon[80661]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-19T18:43:41.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:41 smithi059 ceph-mon[80661]: osdmap e25: 4 total, 3 up, 4 in 2023-12-19T18:43:41.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:41 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:41.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:41 smithi059 ceph-mon[80661]: from='osd.3 [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:43:41.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:41 smithi059 ceph-mon[80661]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:43:41.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:41 smithi059 ceph-mon[80661]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:41.504 INFO:journalctl@ceph.osd.3.smithi089.stdout:Dec 19 18:43:41 smithi089 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-3[93903]: 2023-12-19T18:43:41.037+0000 7f62a5d93700 -1 osd.3 0 waiting for initial osdmap 2023-12-19T18:43:41.505 INFO:journalctl@ceph.osd.3.smithi089.stdout:Dec 19 18:43:41 smithi089 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-3[93903]: 2023-12-19T18:43:41.044+0000 7f62a138f700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T18:43:41.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:41 smithi089 ceph-mon[88459]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-19T18:43:41.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:41 smithi089 ceph-mon[88459]: osdmap e25: 4 total, 3 up, 4 in 2023-12-19T18:43:41.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:41 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:41.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:41 smithi089 ceph-mon[88459]: from='osd.3 [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:43:41.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:41 smithi089 ceph-mon[88459]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:43:41.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:41 smithi089 ceph-mon[88459]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-19T18:43:42.111 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:42 smithi089 ceph-mon[88459]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-12-19T18:43:42.111 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:42 smithi089 ceph-mon[88459]: osdmap e26: 4 total, 3 up, 4 in 2023-12-19T18:43:42.111 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:42 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:42.111 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:42 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:42.111 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:42 smithi089 ceph-mon[88459]: osd.3 [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] boot 2023-12-19T18:43:42.111 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:42 smithi089 ceph-mon[88459]: osdmap e27: 4 total, 4 up, 4 in 2023-12-19T18:43:42.111 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:42 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:42.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:42 smithi059 ceph-mon[80661]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-12-19T18:43:42.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:42 smithi059 ceph-mon[80661]: osdmap e26: 4 total, 3 up, 4 in 2023-12-19T18:43:42.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:42 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:42.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:42 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:42.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:42 smithi059 ceph-mon[80661]: osd.3 [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] boot 2023-12-19T18:43:42.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:42 smithi059 ceph-mon[80661]: osdmap e27: 4 total, 4 up, 4 in 2023-12-19T18:43:42.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:42 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T18:43:42.754 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-19T18:43:42.754 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-19T18:43:42.754 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2023-12-19T18:43:42.754 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2023-12-19T18:43:42.755 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122501 s, 856 MB/s 2023-12-19T18:43:42.755 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2023-12-19T18:43:43.326 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:43 smithi089 ceph-mon[88459]: purged_snaps scrub starts 2023-12-19T18:43:43.326 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:43 smithi089 ceph-mon[88459]: purged_snaps scrub ok 2023-12-19T18:43:43.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:43 smithi089 ceph-mon[88459]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:43:43.327 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:43 smithi089 ceph-mon[88459]: osdmap e28: 4 total, 4 up, 4 in 2023-12-19T18:43:43.352 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_3 2023-12-19T18:43:43.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:43 smithi059 ceph-mon[80661]: purged_snaps scrub starts 2023-12-19T18:43:43.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:43 smithi059 ceph-mon[80661]: purged_snaps scrub ok 2023-12-19T18:43:43.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:43 smithi059 ceph-mon[80661]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:43:43.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:43 smithi059 ceph-mon[80661]: osdmap e28: 4 total, 4 up, 4 in 2023-12-19T18:43:43.784 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi089/config 2023-12-19T18:43:44.934 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: Detected new or changed devices on smithi089 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: Adjusting osd_memory_target on smithi089 to 15936M 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:44.935 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:44.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: osdmap e29: 4 total, 4 up, 4 in 2023-12-19T18:43:44.936 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:44 smithi089 ceph-mon[88459]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:43:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.130+0000 7f2be93d5700 1 -- 172.21.15.89:0/3422925741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 msgr2=0x7f2be40fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:43:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.130+0000 7f2be93d5700 1 --2- 172.21.15.89:0/3422925741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 0x7f2be40fd150 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8004940 tx=0x7f2bd8009e30 comp rx=0 tx=0).stop 2023-12-19T18:43:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.130+0000 7f2be93d5700 1 -- 172.21.15.89:0/3422925741 shutdown_connections 2023-12-19T18:43:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.130+0000 7f2be93d5700 1 --2- 172.21.15.89:0/3422925741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 0x7f2be40fd150 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.130+0000 7f2be93d5700 1 --2- 172.21.15.89:0/3422925741 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be40fbaf0 0x7f2be40fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.130+0000 7f2be93d5700 1 -- 172.21.15.89:0/3422925741 >> 172.21.15.89:0/3422925741 conn(0x7f2be40f7190 msgr2=0x7f2be40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:43:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.130+0000 7f2be93d5700 1 -- 172.21.15.89:0/3422925741 shutdown_connections 2023-12-19T18:43:45.131 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.130+0000 7f2be93d5700 1 -- 172.21.15.89:0/3422925741 wait complete. 2023-12-19T18:43:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.131+0000 7f2be93d5700 1 Processor -- start 2023-12-19T18:43:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.131+0000 7f2be93d5700 1 -- start start 2023-12-19T18:43:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.131+0000 7f2be93d5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be40fbaf0 0x7f2be410c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:43:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.131+0000 7f2be93d5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 0x7f2be410c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:43:45.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.131+0000 7f2be93d5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be410d040 con 0x7f2be40fbaf0 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.131+0000 7f2be93d5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be410d880 con 0x7f2be40fcd00 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.131+0000 7f2be27fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 0x7f2be410c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.131+0000 7f2be27fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 0x7f2be410c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55088/0 (socket says 172.21.15.89:55088) 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2be27fc700 1 -- 172.21.15.89:0/1791082489 learned_addr learned my addr 172.21.15.89:0/1791082489 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2be2ffd700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be40fbaf0 0x7f2be410c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2be27fc700 1 -- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be40fbaf0 msgr2=0x7f2be410c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2be27fc700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be40fbaf0 0x7f2be410c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2be27fc700 1 -- 172.21.15.89:0/1791082489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd8005040 con 0x7f2be40fcd00 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2be27fc700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 0x7f2be410c9b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009360 tx=0x7f2bd8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2bcb7fe700 1 -- 172.21.15.89:0/1791082489 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bd800e040 con 0x7f2be40fcd00 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be410db00 con 0x7f2be40fcd00 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2bcb7fe700 1 -- 172.21.15.89:0/1791082489 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bd800bcd0 con 0x7f2be40fcd00 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2bcb7fe700 1 -- 172.21.15.89:0/1791082489 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bd801c630 con 0x7f2be40fcd00 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.132+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be410e040 con 0x7f2be40fcd00 2023-12-19T18:43:45.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.133+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be404bb80 con 0x7f2be40fcd00 2023-12-19T18:43:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.133+0000 7f2bcb7fe700 1 -- 172.21.15.89:0/1791082489 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2bd8013960 con 0x7f2be40fcd00 2023-12-19T18:43:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.134+0000 7f2bcb7fe700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2bcc068f60 0x7f2bcc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:43:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.134+0000 7f2bcb7fe700 1 -- 172.21.15.89:0/1791082489 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f2bd8087bc0 con 0x7f2be40fcd00 2023-12-19T18:43:45.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.134+0000 7f2be2ffd700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2bcc068f60 0x7f2bcc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:43:45.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.135+0000 7f2be2ffd700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2bcc068f60 0x7f2bcc06b410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4006fd0 tx=0x7f2bd4008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:43:45.139 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.138+0000 7f2bcb7fe700 1 -- 172.21.15.89:0/1791082489 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2bd80581c0 con 0x7f2be40fcd00 2023-12-19T18:43:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: Detected new or changed devices on smithi089 2023-12-19T18:43:45.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: Adjusting osd_memory_target on smithi089 to 15936M 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: osdmap e29: 4 total, 4 up, 4 in 2023-12-19T18:43:45.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:44 smithi059 ceph-mon[80661]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:43:45.332 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:43:45.332+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2be410e460 con 0x7f2bcc068f60 2023-12-19T18:43:45.991 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:45 smithi089 ceph-mon[88459]: osdmap e30: 4 total, 4 up, 4 in 2023-12-19T18:43:45.992 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:45 smithi089 ceph-mon[88459]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:43:45.992 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:43:45.992 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:43:45.992 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:46.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:45 smithi059 ceph-mon[80661]: osdmap e30: 4 total, 4 up, 4 in 2023-12-19T18:43:46.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:45 smithi059 ceph-mon[80661]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:43:46.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:43:46.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:43:46.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:47.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:46 smithi059 ceph-mon[80661]: osdmap e31: 4 total, 4 up, 4 in 2023-12-19T18:43:47.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:46 smithi059 ceph-mon[80661]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:43:47.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:46 smithi089 ceph-mon[88459]: osdmap e31: 4 total, 4 up, 4 in 2023-12-19T18:43:47.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:46 smithi089 ceph-mon[88459]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:43:48.197 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:47 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/154160621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "152f485f-9a72-40e5-9a4d-535e4caf73ca"}]: dispatch 2023-12-19T18:43:48.197 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:47 smithi089 ceph-mon[88459]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "152f485f-9a72-40e5-9a4d-535e4caf73ca"}]: dispatch 2023-12-19T18:43:48.197 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:47 smithi089 ceph-mon[88459]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "152f485f-9a72-40e5-9a4d-535e4caf73ca"}]': finished 2023-12-19T18:43:48.197 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:47 smithi089 ceph-mon[88459]: osdmap e32: 5 total, 4 up, 5 in 2023-12-19T18:43:48.197 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:47 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:43:48.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:47 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/154160621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "152f485f-9a72-40e5-9a4d-535e4caf73ca"}]: dispatch 2023-12-19T18:43:48.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:47 smithi059 ceph-mon[80661]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "152f485f-9a72-40e5-9a4d-535e4caf73ca"}]: dispatch 2023-12-19T18:43:48.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:47 smithi059 ceph-mon[80661]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "152f485f-9a72-40e5-9a4d-535e4caf73ca"}]': finished 2023-12-19T18:43:48.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:47 smithi059 ceph-mon[80661]: osdmap e32: 5 total, 4 up, 5 in 2023-12-19T18:43:48.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:47 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:43:49.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:48 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/72747378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:43:49.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:48 smithi059 ceph-mon[80661]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:43:49.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:48 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/72747378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:43:49.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:48 smithi089 ceph-mon[88459]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:43:51.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:51 smithi089 ceph-mon[88459]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-12-19T18:43:51.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:51 smithi059 ceph-mon[80661]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-12-19T18:43:53.079 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:52 smithi089 ceph-mon[88459]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2023-12-19T18:43:53.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:52 smithi059 ceph-mon[80661]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2023-12-19T18:43:54.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:43:54.179 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:43:55.197 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:54 smithi089 ceph-mon[88459]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-12-19T18:43:55.198 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:54 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-19T18:43:55.198 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:54 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:55.198 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:54 smithi089 ceph-mon[88459]: Deploying daemon osd.4 on smithi089 2023-12-19T18:43:55.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:54 smithi059 ceph-mon[80661]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-12-19T18:43:55.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:54 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-19T18:43:55.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:54 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:43:55.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:54 smithi059 ceph-mon[80661]: Deploying daemon osd.4 on smithi089 2023-12-19T18:43:57.823 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:57 smithi089 ceph-mon[88459]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-12-19T18:43:57.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:57 smithi059 ceph-mon[80661]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-12-19T18:43:59.830 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:59 smithi089 ceph-mon[88459]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-12-19T18:43:59.831 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:59.831 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:59.831 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:43:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:43:59.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:59 smithi059 ceph-mon[80661]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-12-19T18:43:59.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:43:59.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:43:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:44:01.744 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:01 smithi089 ceph-mon[88459]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-12-19T18:44:01.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:01 smithi059 ceph-mon[80661]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-12-19T18:44:02.696 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 4 on host 'smithi089' 2023-12-19T18:44:02.696 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.694+0000 7f2bcb7fe700 1 -- 172.21.15.89:0/1791082489 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2be410e460 con 0x7f2bcc068f60 2023-12-19T18:44:02.697 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2bcc068f60 msgr2=0x7f2bcc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2bcc068f60 0x7f2bcc06b410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4006fd0 tx=0x7f2bd4008040 comp rx=0 tx=0).stop 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 msgr2=0x7f2be410c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 0x7f2be410c9b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009360 tx=0x7f2bd8009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 shutdown_connections 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2bcc068f60 0x7f2bcc06b410 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2be40fbaf0 0x7f2be410c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 --2- 172.21.15.89:0/1791082489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2be40fcd00 0x7f2be410c9b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 >> 172.21.15.89:0/1791082489 conn(0x7f2be40f7190 msgr2=0x7f2be40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 shutdown_connections 2023-12-19T18:44:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:02.697+0000 7f2be93d5700 1 -- 172.21.15.89:0/1791082489 wait complete. 2023-12-19T18:44:03.306 DEBUG:teuthology.orchestra.run.smithi089:osd.4> sudo journalctl -f -n 0 -u ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.4.service 2023-12-19T18:44:03.309 INFO:tasks.cephadm:Deploying osd.5 on smithi089 with /dev/vg_nvme/lv_2... 2023-12-19T18:44:03.309 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-19T18:44:03.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:03 smithi059 ceph-mon[80661]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:44:03.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:03 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:03.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:03 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:03.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:03 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:03.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:03 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:03.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:03 smithi059 ceph-mon[80661]: from='osd.4 [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-19T18:44:03.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:03 smithi059 ceph-mon[80661]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-19T18:44:04.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:03 smithi089 ceph-mon[88459]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:44:04.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:03 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:04.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:03 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:04.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:03 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:04.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:03 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:04.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:03 smithi089 ceph-mon[88459]: from='osd.4 [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-19T18:44:04.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:03 smithi089 ceph-mon[88459]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-19T18:44:04.367 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi089/config 2023-12-19T18:44:05.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:04 smithi059 ceph-mon[80661]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-19T18:44:05.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:04 smithi059 ceph-mon[80661]: osdmap e33: 5 total, 4 up, 5 in 2023-12-19T18:44:05.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:04 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:44:05.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:04 smithi059 ceph-mon[80661]: from='osd.4 [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:44:05.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:04 smithi059 ceph-mon[80661]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:44:05.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:04 smithi059 ceph-mon[80661]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:44:05.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:04 smithi089 ceph-mon[88459]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-19T18:44:05.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:04 smithi089 ceph-mon[88459]: osdmap e33: 5 total, 4 up, 5 in 2023-12-19T18:44:05.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:04 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:44:05.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:04 smithi089 ceph-mon[88459]: from='osd.4 [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:44:05.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:04 smithi089 ceph-mon[88459]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:44:05.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:04 smithi089 ceph-mon[88459]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T18:44:05.255 INFO:journalctl@ceph.osd.4.smithi089.stdout:Dec 19 18:44:04 smithi089 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-4[100755]: 2023-12-19T18:44:04.926+0000 7fd9266d6700 -1 osd.4 0 waiting for initial osdmap 2023-12-19T18:44:05.255 INFO:journalctl@ceph.osd.4.smithi089.stdout:Dec 19 18:44:04 smithi089 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-4[100755]: 2023-12-19T18:44:04.933+0000 7fd9204cf700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T18:44:06.095 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:05 smithi089 ceph-mon[88459]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-12-19T18:44:06.095 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:05 smithi089 ceph-mon[88459]: osdmap e34: 5 total, 4 up, 5 in 2023-12-19T18:44:06.095 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:05 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:44:06.095 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:05 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:44:06.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:05 smithi059 ceph-mon[80661]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-12-19T18:44:06.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:05 smithi059 ceph-mon[80661]: osdmap e34: 5 total, 4 up, 5 in 2023-12-19T18:44:06.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:05 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:44:06.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:05 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:44:06.569 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-19T18:44:06.570 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-19T18:44:06.570 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2023-12-19T18:44:06.570 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2023-12-19T18:44:06.570 INFO:teuthology.orchestra.run.smithi089.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112042 s, 936 MB/s 2023-12-19T18:44:06.570 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: purged_snaps scrub starts 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: purged_snaps scrub ok 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: osd.4 [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] boot 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: osdmap e35: 5 total, 5 up, 5 in 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:07.119 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:07.120 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:07.120 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:44:07.120 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:06 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:07.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: purged_snaps scrub starts 2023-12-19T18:44:07.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: purged_snaps scrub ok 2023-12-19T18:44:07.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: osd.4 [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] boot 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: osdmap e35: 5 total, 5 up, 5 in 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:44:07.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:06 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:07.451 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_2 2023-12-19T18:44:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi089/config 2023-12-19T18:44:08.076 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:07 smithi089 ceph-mon[88459]: Detected new or changed devices on smithi089 2023-12-19T18:44:08.076 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:07 smithi089 ceph-mon[88459]: Adjusting osd_memory_target on smithi089 to 7968M 2023-12-19T18:44:08.076 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:07 smithi089 ceph-mon[88459]: osdmap e36: 5 total, 5 up, 5 in 2023-12-19T18:44:08.076 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:44:08.076 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:08.076 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:44:08.076 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:08.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:07 smithi059 ceph-mon[80661]: Detected new or changed devices on smithi089 2023-12-19T18:44:08.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:07 smithi059 ceph-mon[80661]: Adjusting osd_memory_target on smithi089 to 7968M 2023-12-19T18:44:08.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:07 smithi059 ceph-mon[80661]: osdmap e36: 5 total, 5 up, 5 in 2023-12-19T18:44:08.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:44:08.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:08.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:44:08.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:08.955 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:08 smithi089 ceph-mon[88459]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:08.956 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:08 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:44:09.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 -- 172.21.15.89:0/4139995921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fbaf0 msgr2=0x7f86dc0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:09.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 --2- 172.21.15.89:0/4139995921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fbaf0 0x7f86dc0fbf00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f86d0004940 tx=0x7f86d0009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:09.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 -- 172.21.15.89:0/4139995921 shutdown_connections 2023-12-19T18:44:09.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 --2- 172.21.15.89:0/4139995921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86dc0fcd00 0x7f86dc0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:09.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 --2- 172.21.15.89:0/4139995921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fbaf0 0x7f86dc0fbf00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:09.133 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 -- 172.21.15.89:0/4139995921 >> 172.21.15.89:0/4139995921 conn(0x7f86dc0f71d0 msgr2=0x7f86dc0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 -- 172.21.15.89:0/4139995921 shutdown_connections 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 -- 172.21.15.89:0/4139995921 wait complete. 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 Processor -- start 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.131+0000 7f86e2ed2700 1 -- start start 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86e2ed2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86dc0fbaf0 0x7f86dc10d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86e2ed2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fcd00 0x7f86dc10dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86e2ed2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc10e170 con 0x7f86dc0fbaf0 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86e2ed2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc10e9b0 con 0x7f86dc0fcd00 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86dbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fcd00 0x7f86dc10dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86dbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fcd00 0x7f86dc10dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34000/0 (socket says 172.21.15.89:34000) 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86dbfff700 1 -- 172.21.15.89:0/2344728262 learned_addr learned my addr 172.21.15.89:0/2344728262 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86e0c6e700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86dc0fbaf0 0x7f86dc10d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86dbfff700 1 -- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86dc0fbaf0 msgr2=0x7f86dc10d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86dbfff700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86dc0fbaf0 0x7f86dc10d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:09.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86dbfff700 1 -- 172.21.15.89:0/2344728262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0005040 con 0x7f86dc0fcd00 2023-12-19T18:44:09.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.132+0000 7f86dbfff700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fcd00 0x7f86dc10dae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b790 tx=0x7f86c8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:09.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.133+0000 7f86d97fa700 1 -- 172.21.15.89:0/2344728262 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86c800f770 con 0x7f86dc0fcd00 2023-12-19T18:44:09.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.133+0000 7f86d97fa700 1 -- 172.21.15.89:0/2344728262 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86c800fd80 con 0x7f86dc0fcd00 2023-12-19T18:44:09.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.133+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc10ec90 con 0x7f86dc0fcd00 2023-12-19T18:44:09.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.133+0000 7f86d97fa700 1 -- 172.21.15.89:0/2344728262 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86c800d310 con 0x7f86dc0fcd00 2023-12-19T18:44:09.135 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.133+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc10f1d0 con 0x7f86dc0fcd00 2023-12-19T18:44:09.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.134+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc04bb80 con 0x7f86dc0fcd00 2023-12-19T18:44:09.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.135+0000 7f86d97fa700 1 -- 172.21.15.89:0/2344728262 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f86c8010420 con 0x7f86dc0fcd00 2023-12-19T18:44:09.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.135+0000 7f86d97fa700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f86cc068f60 0x7f86cc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:09.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.135+0000 7f86d97fa700 1 -- 172.21.15.89:0/2344728262 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f86c8086770 con 0x7f86dc0fcd00 2023-12-19T18:44:09.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.135+0000 7f86e0c6e700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f86cc068f60 0x7f86cc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:09.136 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.136+0000 7f86e0c6e700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f86cc068f60 0x7f86cc06b410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f86d000b3b0 tx=0x7f86d0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:09.143 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.143+0000 7f86d97fa700 1 -- 172.21.15.89:0/2344728262 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f86c8056c60 con 0x7f86dc0fcd00 2023-12-19T18:44:09.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:08 smithi059 ceph-mon[80661]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:09.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:44:09.344 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:09.343+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f86dc10f730 con 0x7f86cc068f60 2023-12-19T18:44:10.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:09 smithi089 ceph-mon[88459]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:44:10.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:44:10.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:44:10.015 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:10.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:09 smithi059 ceph-mon[80661]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:44:10.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T18:44:10.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T18:44:10.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:11.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:10 smithi059 ceph-mon[80661]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:11.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:10 smithi089 ceph-mon[88459]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:12.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:11 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/878455107' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92c1e7d-9b7c-4594-b8db-7f812da2afc6"}]: dispatch 2023-12-19T18:44:12.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:11 smithi059 ceph-mon[80661]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92c1e7d-9b7c-4594-b8db-7f812da2afc6"}]: dispatch 2023-12-19T18:44:12.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:11 smithi059 ceph-mon[80661]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a92c1e7d-9b7c-4594-b8db-7f812da2afc6"}]': finished 2023-12-19T18:44:12.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:11 smithi059 ceph-mon[80661]: osdmap e37: 6 total, 5 up, 6 in 2023-12-19T18:44:12.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:11 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:12.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:11 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/878455107' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92c1e7d-9b7c-4594-b8db-7f812da2afc6"}]: dispatch 2023-12-19T18:44:12.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:11 smithi089 ceph-mon[88459]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a92c1e7d-9b7c-4594-b8db-7f812da2afc6"}]: dispatch 2023-12-19T18:44:12.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:11 smithi089 ceph-mon[88459]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a92c1e7d-9b7c-4594-b8db-7f812da2afc6"}]': finished 2023-12-19T18:44:12.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:11 smithi089 ceph-mon[88459]: osdmap e37: 6 total, 5 up, 6 in 2023-12-19T18:44:12.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:11 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:13.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:12 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/3404639567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:44:13.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:12 smithi059 ceph-mon[80661]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:13.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:12 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/3404639567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T18:44:13.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:12 smithi089 ceph-mon[88459]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:15.701 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:15 smithi089 ceph-mon[88459]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:15.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:15 smithi059 ceph-mon[80661]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:17.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:17 smithi089 ceph-mon[88459]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:17.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:17 smithi059 ceph-mon[80661]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:19.846 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:19 smithi089 ceph-mon[88459]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:19.847 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:19 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-19T18:44:19.847 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:19 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:19.847 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:19 smithi089 ceph-mon[88459]: Deploying daemon osd.5 on smithi089 2023-12-19T18:44:19.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:19 smithi059 ceph-mon[80661]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:19.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:19 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-19T18:44:19.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:19 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:19.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:19 smithi059 ceph-mon[80661]: Deploying daemon osd.5 on smithi089 2023-12-19T18:44:21.752 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:21 smithi059 ceph-mon[80661]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:21.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:21 smithi089 ceph-mon[88459]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:23.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:22 smithi059 ceph-mon[80661]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:23.247 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:22 smithi089 ceph-mon[88459]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:24.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:24.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:24.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:44:24.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:24 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:44:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:24.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:44:24.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:24 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:44:25.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:25 smithi089 ceph-mon[88459]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:25.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:25 smithi059 ceph-mon[80661]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:27.362 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.361+0000 7f86d97fa700 1 -- 172.21.15.89:0/2344728262 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f86dc10f730 con 0x7f86cc068f60 2023-12-19T18:44:27.362 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 5 on host 'smithi089' 2023-12-19T18:44:27.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.363+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f86cc068f60 msgr2=0x7f86cc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:27.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.363+0000 7f86e2ed2700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f86cc068f60 0x7f86cc06b410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f86d000b3b0 tx=0x7f86d0009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:27.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.363+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fcd00 msgr2=0x7f86dc10dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:27.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.363+0000 7f86e2ed2700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fcd00 0x7f86dc10dae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b790 tx=0x7f86c8009250 comp rx=0 tx=0).stop 2023-12-19T18:44:27.364 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.364+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 shutdown_connections 2023-12-19T18:44:27.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.364+0000 7f86e2ed2700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f86cc068f60 0x7f86cc06b410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:27.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.364+0000 7f86e2ed2700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86dc0fbaf0 0x7f86dc10d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:27.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.364+0000 7f86e2ed2700 1 --2- 172.21.15.89:0/2344728262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86dc0fcd00 0x7f86dc10dae0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:27.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.364+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 >> 172.21.15.89:0/2344728262 conn(0x7f86dc0f71d0 msgr2=0x7f86dc065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:27.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.364+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 shutdown_connections 2023-12-19T18:44:27.365 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:27.364+0000 7f86e2ed2700 1 -- 172.21.15.89:0/2344728262 wait complete. 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='osd.5 [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.814 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:27 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='osd.5 [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:27.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:27 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:44:28.088 DEBUG:teuthology.orchestra.run.smithi089:osd.5> sudo journalctl -f -n 0 -u ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.5.service 2023-12-19T18:44:28.090 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2023-12-19T18:44:28.090 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-19T18:44:28.368 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:29.185 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:29 smithi089 ceph-mon[88459]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-19T18:44:29.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:29 smithi089 ceph-mon[88459]: osdmap e38: 6 total, 5 up, 6 in 2023-12-19T18:44:29.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:29 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:29.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:29 smithi089 ceph-mon[88459]: from='osd.5 [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:44:29.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:29 smithi089 ceph-mon[88459]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:44:29.186 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:29 smithi089 ceph-mon[88459]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:29.186 INFO:journalctl@ceph.osd.5.smithi089.stdout:Dec 19 18:44:29 smithi089 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-5[107486]: 2023-12-19T18:44:29.037+0000 7f34e7f75700 -1 osd.5 0 waiting for initial osdmap 2023-12-19T18:44:29.186 INFO:journalctl@ceph.osd.5.smithi089.stdout:Dec 19 18:44:29 smithi089 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-5[107486]: 2023-12-19T18:44:29.044+0000 7f34e3571700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T18:44:29.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:29 smithi059 ceph-mon[80661]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-19T18:44:29.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:29 smithi059 ceph-mon[80661]: osdmap e38: 6 total, 5 up, 6 in 2023-12-19T18:44:29.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:29 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:29.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:29 smithi059 ceph-mon[80661]: from='osd.5 [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:44:29.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:29 smithi059 ceph-mon[80661]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2023-12-19T18:44:29.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:29 smithi059 ceph-mon[80661]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T18:44:30.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.114+0000 7f78eb550700 1 -- 172.21.15.59:0/1922503512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fbaf0 msgr2=0x7f78e40fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:30.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.114+0000 7f78eb550700 1 --2- 172.21.15.59:0/1922503512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fbaf0 0x7f78e40fbf00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f78e0004940 tx=0x7f78e0009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:30.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.114+0000 7f78eb550700 1 -- 172.21.15.59:0/1922503512 shutdown_connections 2023-12-19T18:44:30.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.114+0000 7f78eb550700 1 --2- 172.21.15.59:0/1922503512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40fcd00 0x7f78e40fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.114+0000 7f78eb550700 1 --2- 172.21.15.59:0/1922503512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fbaf0 0x7f78e40fbf00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.114+0000 7f78eb550700 1 -- 172.21.15.59:0/1922503512 >> 172.21.15.59:0/1922503512 conn(0x7f78e40f7190 msgr2=0x7f78e40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.114+0000 7f78eb550700 1 -- 172.21.15.59:0/1922503512 shutdown_connections 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.114+0000 7f78eb550700 1 -- 172.21.15.59:0/1922503512 wait complete. 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78eb550700 1 Processor -- start 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78eb550700 1 -- start start 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78eb550700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40fbaf0 0x7f78e41102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78eb550700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fcd00 0x7f78e41127f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78eb550700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e4066110 con 0x7f78e40fcd00 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78eb550700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e4066250 con 0x7f78e40fbaf0 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78e8aeb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fcd00 0x7f78e41127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78e8aeb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fcd00 0x7f78e41127f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36120/0 (socket says 172.21.15.59:36120) 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78e8aeb700 1 -- 172.21.15.59:0/1451185148 learned_addr learned my addr 172.21.15.59:0/1451185148 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:30.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78e92ec700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40fbaf0 0x7f78e41102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:30.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78e8aeb700 1 -- 172.21.15.59:0/1451185148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40fbaf0 msgr2=0x7f78e41102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:30.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78e8aeb700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40fbaf0 0x7f78e41102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:30.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.115+0000 7f78e8aeb700 1 -- 172.21.15.59:0/1451185148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e0005040 con 0x7f78e40fcd00 2023-12-19T18:44:30.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.116+0000 7f78e8aeb700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fcd00 0x7f78e41127f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f78d4002a50 tx=0x7f78d40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:30.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.116+0000 7f78d9ffb700 1 -- 172.21.15.59:0/1451185148 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78d4007cc0 con 0x7f78e40fcd00 2023-12-19T18:44:30.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.116+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e4112df0 con 0x7f78e40fcd00 2023-12-19T18:44:30.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.116+0000 7f78d9ffb700 1 -- 172.21.15.59:0/1451185148 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78d400f040 con 0x7f78e40fcd00 2023-12-19T18:44:30.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.116+0000 7f78d9ffb700 1 -- 172.21.15.59:0/1451185148 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78d400d9c0 con 0x7f78e40fcd00 2023-12-19T18:44:30.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.116+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e41132b0 con 0x7f78e40fcd00 2023-12-19T18:44:30.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.117+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e404bb80 con 0x7f78e40fcd00 2023-12-19T18:44:30.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.117+0000 7f78d9ffb700 1 -- 172.21.15.59:0/1451185148 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f78d400dc40 con 0x7f78e40fcd00 2023-12-19T18:44:30.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.118+0000 7f78d9ffb700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78d0068f60 0x7f78d006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:30.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.118+0000 7f78d9ffb700 1 -- 172.21.15.59:0/1451185148 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f78d40867e0 con 0x7f78e40fcd00 2023-12-19T18:44:30.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.118+0000 7f78e92ec700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78d0068f60 0x7f78d006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:30.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.118+0000 7f78e92ec700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78d0068f60 0x7f78d006b410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78e000b7d0 tx=0x7f78e0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:30.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.122+0000 7f78d9ffb700 1 -- 172.21.15.59:0/1451185148 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f78d4052f70 con 0x7f78e40fcd00 2023-12-19T18:44:30.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:30 smithi059 ceph-mon[80661]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-12-19T18:44:30.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:30 smithi059 ceph-mon[80661]: osdmap e39: 6 total, 5 up, 6 in 2023-12-19T18:44:30.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:30 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:30.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:30 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:30.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:30 smithi089 ceph-mon[88459]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2023-12-19T18:44:30.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:30 smithi089 ceph-mon[88459]: osdmap e39: 6 total, 5 up, 6 in 2023-12-19T18:44:30.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:30 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:30.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:30 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:30.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.325+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f78e4104490 con 0x7f78e40fcd00 2023-12-19T18:44:30.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.325+0000 7f78d9ffb700 1 -- 172.21.15.59:0/1451185148 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f78d4056590 con 0x7f78e40fcd00 2023-12-19T18:44:30.326 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:44:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78d0068f60 msgr2=0x7f78d006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78d0068f60 0x7f78d006b410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78e000b7d0 tx=0x7f78e0009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fcd00 msgr2=0x7f78e41127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fcd00 0x7f78e41127f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f78d4002a50 tx=0x7f78d40046b0 comp rx=0 tx=0).stop 2023-12-19T18:44:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 shutdown_connections 2023-12-19T18:44:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78e40fbaf0 0x7f78e41102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:30.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78d0068f60 0x7f78d006b410 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 --2- 172.21.15.59:0/1451185148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78e40fcd00 0x7f78e41127f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 >> 172.21.15.59:0/1451185148 conn(0x7f78e40f7190 msgr2=0x7f78e40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.327+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 shutdown_connections 2023-12-19T18:44:30.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:30.328+0000 7f78eb550700 1 -- 172.21.15.59:0/1451185148 wait complete. 2023-12-19T18:44:31.099 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1703011470,"num_in_osds":6,"osd_in_since":1703011451,"num_remapped_pgs":0} 2023-12-19T18:44:31.100 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-19T18:44:31.366 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:31.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: purged_snaps scrub starts 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: purged_snaps scrub ok 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: osd.5 [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] boot 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: osdmap e40: 6 total, 6 up, 6 in 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1451185148' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: Detected new or changed devices on smithi089 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:31.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:31.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:31.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: Adjusting osd_memory_target on smithi089 to 5312M 2023-12-19T18:44:31.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:31.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:31.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:44:31.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:31 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:31.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: purged_snaps scrub starts 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: purged_snaps scrub ok 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: osd.5 [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] boot 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: osdmap e40: 6 total, 6 up, 6 in 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1451185148' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: Detected new or changed devices on smithi089 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-19T18:44:31.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: Adjusting osd_memory_target on smithi089 to 5312M 2023-12-19T18:44:31.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:31.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:44:31.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:44:31.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:31 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:44:32.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:32 smithi059 ceph-mon[80661]: osdmap e41: 6 total, 6 up, 6 in 2023-12-19T18:44:32.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:32 smithi089 ceph-mon[88459]: osdmap e41: 6 total, 6 up, 6 in 2023-12-19T18:44:33.127 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:33 smithi059 ceph-mon[80661]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:33.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.161+0000 7f7471b3c700 1 -- 172.21.15.59:0/354495038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 msgr2=0x7f746c0fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:33.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.161+0000 7f7471b3c700 1 --2- 172.21.15.59:0/354495038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 0x7f746c0fcc10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7460004940 tx=0x7f7460009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:33.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.161+0000 7f7471b3c700 1 -- 172.21.15.59:0/354495038 shutdown_connections 2023-12-19T18:44:33.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.162+0000 7f7471b3c700 1 --2- 172.21.15.59:0/354495038 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 0x7f746c0fcc10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:33.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.162+0000 7f7471b3c700 1 --2- 172.21.15.59:0/354495038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c102c00 0x7f746c102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:33.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.162+0000 7f7471b3c700 1 -- 172.21.15.59:0/354495038 >> 172.21.15.59:0/354495038 conn(0x7f746c0f8790 msgr2=0x7f746c0faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:33.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.162+0000 7f7471b3c700 1 -- 172.21.15.59:0/354495038 shutdown_connections 2023-12-19T18:44:33.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.162+0000 7f7471b3c700 1 -- 172.21.15.59:0/354495038 wait complete. 2023-12-19T18:44:33.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.162+0000 7f7471b3c700 1 Processor -- start 2023-12-19T18:44:33.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.162+0000 7f7471b3c700 1 -- start start 2023-12-19T18:44:33.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f7471b3c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c102c00 0x7f746c1135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:33.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f7471b3c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 0x7f746c113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:33.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f7471b3c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746c114130 con 0x7f746c1035a0 2023-12-19T18:44:33.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f7471b3c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746c10d4d0 con 0x7f746c102c00 2023-12-19T18:44:33.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f746affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 0x7f746c113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:33.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f746affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 0x7f746c113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34242/0 (socket says 172.21.15.59:34242) 2023-12-19T18:44:33.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f746affd700 1 -- 172.21.15.59:0/3816994560 learned_addr learned my addr 172.21.15.59:0/3816994560 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:33.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f746b7fe700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c102c00 0x7f746c1135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:33.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f746affd700 1 -- 172.21.15.59:0/3816994560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c102c00 msgr2=0x7f746c1135a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:33.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f746affd700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c102c00 0x7f746c1135a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:33.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f746affd700 1 -- 172.21.15.59:0/3816994560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7460005040 con 0x7f746c1035a0 2023-12-19T18:44:33.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f746affd700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 0x7f746c113ae0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7460009fd0 tx=0x7f7460009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:33.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f7453fff700 1 -- 172.21.15.59:0/3816994560 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f746000e040 con 0x7f746c1035a0 2023-12-19T18:44:33.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746c10d750 con 0x7f746c1035a0 2023-12-19T18:44:33.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f7453fff700 1 -- 172.21.15.59:0/3816994560 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7460013420 con 0x7f746c1035a0 2023-12-19T18:44:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f7453fff700 1 -- 172.21.15.59:0/3816994560 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f746001c630 con 0x7f746c1035a0 2023-12-19T18:44:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.163+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746c10dc90 con 0x7f746c1035a0 2023-12-19T18:44:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.164+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746c1050d0 con 0x7f746c1035a0 2023-12-19T18:44:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.165+0000 7f7453fff700 1 -- 172.21.15.59:0/3816994560 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7460013a00 con 0x7f746c1035a0 2023-12-19T18:44:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.165+0000 7f7453fff700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7454068f60 0x7f745406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.165+0000 7f7453fff700 1 -- 172.21.15.59:0/3816994560 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7460087c40 con 0x7f746c1035a0 2023-12-19T18:44:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.165+0000 7f746b7fe700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7454068f60 0x7f745406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.166+0000 7f746b7fe700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7454068f60 0x7f745406b410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f745c0041d0 tx=0x7f745c0043c0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:33.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.170+0000 7f7453fff700 1 -- 172.21.15.59:0/3816994560 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7460057ee0 con 0x7f746c1035a0 2023-12-19T18:44:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.365+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f746c109b90 con 0x7f746c1035a0 2023-12-19T18:44:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.366+0000 7f7453fff700 1 -- 172.21.15.59:0/3816994560 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11030 (secure 0 0 0) 0x7f7460057a70 con 0x7f746c1035a0 2023-12-19T18:44:33.367 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:44:33.367 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":41,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","created":"2023-12-19T18:38:57.167145+0000","modified":"2023-12-19T18:44:31.038588+0000","last_up_change":"2023-12-19T18:44:30.032151+0000","last_in_change":"2023-12-19T18:44:11.480055+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T18:43:21.014480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a33e1310-4169-41e4-bd1f-748898ef9289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6803","nonce":177591080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6805","nonce":177591080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6809","nonce":177591080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6807","nonce":177591080}]},"public_addr":"172.21.15.59:6803/177591080","cluster_addr":"172.21.15.59:6805/177591080","heartbeat_back_addr":"172.21.15.59:6809/177591080","heartbeat_front_addr":"172.21.15.59:6807/177591080","state":["exists","up"]},{"osd":1,"uuid":"73a2cb1c-90fe-4d56-adfd-78a688c6b2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6811","nonce":1239732570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6813","nonce":1239732570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6817","nonce":1239732570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6815","nonce":1239732570}]},"public_addr":"172.21.15.59:6811/1239732570","cluster_addr":"172.21.15.59:6813/1239732570","heartbeat_back_addr":"172.21.15.59:6817/1239732570","heartbeat_front_addr":"172.21.15.59:6815/1239732570","state":["exists","up"]},{"osd":2,"uuid":"0003bcc2-1e16-44d8-a031-8e26fa6c7467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6819","nonce":240742436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6821","nonce":240742436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6825","nonce":240742436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6823","nonce":240742436}]},"public_addr":"172.21.15.59:6819/240742436","cluster_addr":"172.21.15.59:6821/240742436","heartbeat_back_addr":"172.21.15.59:6825/240742436","heartbeat_front_addr":"172.21.15.59:6823/240742436","state":["exists","up"]},{"osd":3,"uuid":"08b40954-a1b3-4754-8494-e3a5423406c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6801","nonce":2655128462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6803","nonce":2655128462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6807","nonce":2655128462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6805","nonce":2655128462}]},"public_addr":"172.21.15.89:6801/2655128462","cluster_addr":"172.21.15.89:6803/2655128462","heartbeat_back_addr":"172.21.15.89:6807/2655128462","heartbeat_front_addr":"172.21.15.89:6805/2655128462","state":["exists","up"]},{"osd":4,"uuid":"152f485f-9a72-40e5-9a4d-535e4caf73ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6809","nonce":2501804384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6811","nonce":2501804384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6815","nonce":2501804384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6813","nonce":2501804384}]},"public_addr":"172.21.15.89:6809/2501804384","cluster_addr":"172.21.15.89:6811/2501804384","heartbeat_back_addr":"172.21.15.89:6815/2501804384","heartbeat_front_addr":"172.21.15.89:6813/2501804384","state":["exists","up"]},{"osd":5,"uuid":"a92c1e7d-9b7c-4594-b8db-7f812da2afc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6817","nonce":1872882017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6819","nonce":1872882017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6823","nonce":1872882017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6821","nonce":1872882017}]},"public_addr":"172.21.15.89:6817/1872882017","cluster_addr":"172.21.15.89:6819/1872882017","heartbeat_back_addr":"172.21.15.89:6823/1872882017","heartbeat_front_addr":"172.21.15.89:6821/1872882017","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:42:23.049328+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:42:50.671474+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:43:17.816383+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:43:40.185718+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:44:04.182008+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:44:28.308577+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/3325232385":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/3905471834":"2023-12-20T18:39:58.643904+0000","172.21.15.59:6801/6145510":"2023-12-20T18:41:08.486998+0000","172.21.15.59:6801/2067989811":"2023-12-20T18:39:58.643904+0000","172.21.15.59:6800/1973265269":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/3575411648":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/1003305302":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/3060878312":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/1665894963":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/990606911":"2023-12-20T18:39:27.397002+0000","172.21.15.59:6801/1973265269":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/1707232246":"2023-12-20T18:39:27.397002+0000","172.21.15.59:6800/6145510":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/4127123819":"2023-12-20T18:41:08.486998+0000","172.21.15.59:6800/2067989811":"2023-12-20T18:39:58.643904+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7454068f60 msgr2=0x7f745406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7454068f60 0x7f745406b410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f745c0041d0 tx=0x7f745c0043c0 comp rx=0 tx=0).stop 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 msgr2=0x7f746c113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 0x7f746c113ae0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7460009fd0 tx=0x7f7460009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 shutdown_connections 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f746c102c00 0x7f746c1135a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7454068f60 0x7f745406b410 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 --2- 172.21.15.59:0/3816994560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f746c1035a0 0x7f746c113ae0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 >> 172.21.15.59:0/3816994560 conn(0x7f746c0f8790 msgr2=0x7f746c0fa330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:33.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 shutdown_connections 2023-12-19T18:44:33.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:33.368+0000 7f7471b3c700 1 -- 172.21.15.59:0/3816994560 wait complete. 2023-12-19T18:44:33.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:33 smithi089 ceph-mon[88459]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:34.033 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T18:43:21.014480+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T18:44:34.033 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd pool get .mgr pg_num 2023-12-19T18:44:34.306 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:34.427 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:34 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3816994560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T18:44:34.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:34 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3816994560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T18:44:35.309 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:35 smithi089 ceph-mon[88459]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:35.320 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:35 smithi059 ceph-mon[80661]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.911+0000 7f5118682700 1 -- 172.21.15.59:0/2912758478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51100fb1e0 msgr2=0x7f51100fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.911+0000 7f5118682700 1 --2- 172.21.15.59:0/2912758478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51100fb1e0 0x7f51100fb630 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5100004940 tx=0x7f5100009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:35.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.911+0000 7f5118682700 1 -- 172.21.15.59:0/2912758478 shutdown_connections 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.911+0000 7f5118682700 1 --2- 172.21.15.59:0/2912758478 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f51100fb1e0 0x7f51100fb630 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.911+0000 7f5118682700 1 --2- 172.21.15.59:0/2912758478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5110102c70 0x7f5110103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.911+0000 7f5118682700 1 -- 172.21.15.59:0/2912758478 >> 172.21.15.59:0/2912758478 conn(0x7f51100711a0 msgr2=0x7f51100715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.911+0000 7f5118682700 1 -- 172.21.15.59:0/2912758478 shutdown_connections 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.911+0000 7f5118682700 1 -- 172.21.15.59:0/2912758478 wait complete. 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5118682700 1 Processor -- start 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5118682700 1 -- start start 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5118682700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51100fb1e0 0x7f51100fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5118682700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110102c70 0x7f51100fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5118682700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51100fcf00 con 0x7f5110102c70 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5118682700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51100fd070 con 0x7f51100fb1e0 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5115c1d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110102c70 0x7f51100fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5115c1d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110102c70 0x7f51100fc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34256/0 (socket says 172.21.15.59:34256) 2023-12-19T18:44:35.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5115c1d700 1 -- 172.21.15.59:0/1528734916 learned_addr learned my addr 172.21.15.59:0/1528734916 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f511641e700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51100fb1e0 0x7f51100fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5115c1d700 1 -- 172.21.15.59:0/1528734916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51100fb1e0 msgr2=0x7f51100fc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5115c1d700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51100fb1e0 0x7f51100fc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5115c1d700 1 -- 172.21.15.59:0/1528734916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5100005040 con 0x7f5110102c70 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.912+0000 7f5115c1d700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110102c70 0x7f51100fc900 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f510000b3b0 tx=0x7f5100009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.913+0000 7f5106ffd700 1 -- 172.21.15.59:0/1528734916 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f510000e040 con 0x7f5110102c70 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.913+0000 7f5106ffd700 1 -- 172.21.15.59:0/1528734916 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f510001c430 con 0x7f5110102c70 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.913+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5110119870 con 0x7f5110102c70 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.913+0000 7f5106ffd700 1 -- 172.21.15.59:0/1528734916 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5100013640 con 0x7f5110102c70 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.913+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5110119cd0 con 0x7f5110102c70 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.914+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5110105050 con 0x7f5110102c70 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.914+0000 7f5106ffd700 1 -- 172.21.15.59:0/1528734916 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f510001c5a0 con 0x7f5110102c70 2023-12-19T18:44:35.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.915+0000 7f5106ffd700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f50fc068f60 0x7f50fc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.915+0000 7f5106ffd700 1 -- 172.21.15.59:0/1528734916 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5100087590 con 0x7f5110102c70 2023-12-19T18:44:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.915+0000 7f511641e700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f50fc068f60 0x7f50fc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.915+0000 7f511641e700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f50fc068f60 0x7f50fc06b410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f510c004f40 tx=0x7f510c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:35.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:35.919+0000 7f5106ffd700 1 -- 172.21.15.59:0/1528734916 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5100053e50 con 0x7f5110102c70 2023-12-19T18:44:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.113+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f51100fdb00 con 0x7f5110102c70 2023-12-19T18:44:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.114+0000 7f5106ffd700 1 -- 172.21.15.59:0/1528734916 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5100057470 con 0x7f5110102c70 2023-12-19T18:44:36.115 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2023-12-19T18:44:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f50fc068f60 msgr2=0x7f50fc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f50fc068f60 0x7f50fc06b410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f510c004f40 tx=0x7f510c00b040 comp rx=0 tx=0).stop 2023-12-19T18:44:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110102c70 msgr2=0x7f51100fc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110102c70 0x7f51100fc900 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f510000b3b0 tx=0x7f5100009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 shutdown_connections 2023-12-19T18:44:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51100fb1e0 0x7f51100fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f50fc068f60 0x7f50fc06b410 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 --2- 172.21.15.59:0/1528734916 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5110102c70 0x7f51100fc900 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 >> 172.21.15.59:0/1528734916 conn(0x7f51100711a0 msgr2=0x7f51100fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 shutdown_connections 2023-12-19T18:44:36.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:36.116+0000 7f5118682700 1 -- 172.21.15.59:0/1528734916 wait complete. 2023-12-19T18:44:36.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:36 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1528734916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-19T18:44:36.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:36 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1528734916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-19T18:44:36.956 INFO:tasks.cephadm:Setting up client nodes... 2023-12-19T18:44:36.957 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-19T18:44:37.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:37 smithi059 ceph-mon[80661]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:37.229 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:37.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:37 smithi089 ceph-mon[88459]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.771+0000 7fe179c9f700 1 -- 172.21.15.59:0/4039107682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 msgr2=0x7fe1740fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.771+0000 7fe179c9f700 1 --2- 172.21.15.59:0/4039107682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 0x7fe1740fe3b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe168004940 tx=0x7fe168009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.771+0000 7fe179c9f700 1 -- 172.21.15.59:0/4039107682 shutdown_connections 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.771+0000 7fe179c9f700 1 --2- 172.21.15.59:0/4039107682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 0x7fe1740fe3b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.771+0000 7fe179c9f700 1 --2- 172.21.15.59:0/4039107682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1740faed0 0x7fe1740fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.771+0000 7fe179c9f700 1 -- 172.21.15.59:0/4039107682 >> 172.21.15.59:0/4039107682 conn(0x7fe1740f6950 msgr2=0x7fe1740f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.772+0000 7fe179c9f700 1 -- 172.21.15.59:0/4039107682 shutdown_connections 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.772+0000 7fe179c9f700 1 -- 172.21.15.59:0/4039107682 wait complete. 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.772+0000 7fe179c9f700 1 Processor -- start 2023-12-19T18:44:38.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.772+0000 7fe179c9f700 1 -- start start 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.772+0000 7fe179c9f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1740faed0 0x7fe17410d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.772+0000 7fe179c9f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 0x7fe17410db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.772+0000 7fe179c9f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17410e190 con 0x7fe1740fdf60 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.772+0000 7fe179c9f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17410ea30 con 0x7fe1740faed0 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe172ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 0x7fe17410db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe172ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 0x7fe17410db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34280/0 (socket says 172.21.15.59:34280) 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe172ffd700 1 -- 172.21.15.59:0/3040026217 learned_addr learned my addr 172.21.15.59:0/3040026217 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe1737fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1740faed0 0x7fe17410d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe172ffd700 1 -- 172.21.15.59:0/3040026217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1740faed0 msgr2=0x7fe17410d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe172ffd700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1740faed0 0x7fe17410d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe172ffd700 1 -- 172.21.15.59:0/3040026217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe168005040 con 0x7fe1740fdf60 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe172ffd700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 0x7fe17410db90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe1680093b0 tx=0x7fe168009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe15bfff700 1 -- 172.21.15.59:0/3040026217 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe16800e040 con 0x7fe1740fdf60 2023-12-19T18:44:38.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe15bfff700 1 -- 172.21.15.59:0/3040026217 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe168013420 con 0x7fe1740fdf60 2023-12-19T18:44:38.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17410ecb0 con 0x7fe1740fdf60 2023-12-19T18:44:38.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe15bfff700 1 -- 172.21.15.59:0/3040026217 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe16801c630 con 0x7fe1740fdf60 2023-12-19T18:44:38.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.773+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17410f1f0 con 0x7fe1740fdf60 2023-12-19T18:44:38.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.774+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1741075f0 con 0x7fe1740fdf60 2023-12-19T18:44:38.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.775+0000 7fe15bfff700 1 -- 172.21.15.59:0/3040026217 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe16800bb80 con 0x7fe1740fdf60 2023-12-19T18:44:38.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.775+0000 7fe15bfff700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe15c068f60 0x7fe15c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:38.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.775+0000 7fe15bfff700 1 -- 172.21.15.59:0/3040026217 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe168087d90 con 0x7fe1740fdf60 2023-12-19T18:44:38.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.775+0000 7fe1737fe700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe15c068f60 0x7fe15c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:38.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.776+0000 7fe1737fe700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe15c068f60 0x7fe15c06b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe174065960 tx=0x7fe164009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:38.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:38.780+0000 7fe15bfff700 1 -- 172.21.15.59:0/3040026217 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe168058030 con 0x7fe1740fdf60 2023-12-19T18:44:38.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:44:39.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:44:39.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.061+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fe17404bb80 con 0x7fe1740fdf60 2023-12-19T18:44:39.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.067+0000 7fe15bfff700 1 -- 172.21.15.59:0/3040026217 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fe168057bc0 con 0x7fe1740fdf60 2023-12-19T18:44:39.068 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2023-12-19T18:44:39.068 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQCX5IFly227AxAACSF2R1vMCSisd2iHB3NQ7A== 2023-12-19T18:44:39.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe15c068f60 msgr2=0x7fe15c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:39.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe15c068f60 0x7fe15c06b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe174065960 tx=0x7fe164009040 comp rx=0 tx=0).stop 2023-12-19T18:44:39.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 msgr2=0x7fe17410db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 0x7fe17410db90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe1680093b0 tx=0x7fe168009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 shutdown_connections 2023-12-19T18:44:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1740faed0 0x7fe17410d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe15c068f60 0x7fe15c06b410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:39.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 --2- 172.21.15.59:0/3040026217 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1740fdf60 0x7fe17410db90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:39.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 >> 172.21.15.59:0/3040026217 conn(0x7fe1740f6950 msgr2=0x7fe1740f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:39.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 shutdown_connections 2023-12-19T18:44:39.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:39.069+0000 7fe179c9f700 1 -- 172.21.15.59:0/3040026217 wait complete. 2023-12-19T18:44:39.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:39 smithi089 ceph-mon[88459]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:39.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:39 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3040026217' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T18:44:39.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:39 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3040026217' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-19T18:44:39.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:39 smithi059 ceph-mon[80661]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:39.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:39 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3040026217' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T18:44:39.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:39 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3040026217' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-19T18:44:39.994 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:44:39.994 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-12-19T18:44:39.994 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-19T18:44:40.047 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-19T18:44:40.310 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi089/config 2023-12-19T18:44:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.819+0000 7f0f6a297700 1 -- 172.21.15.89:0/1681715654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 msgr2=0x7f0f640fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.819+0000 7f0f6a297700 1 --2- 172.21.15.89:0/1681715654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 0x7f0f640fe3b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c004940 tx=0x7f0f4c009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.819+0000 7f0f6a297700 1 -- 172.21.15.89:0/1681715654 shutdown_connections 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.819+0000 7f0f6a297700 1 --2- 172.21.15.89:0/1681715654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 0x7f0f640fe3b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.819+0000 7f0f6a297700 1 --2- 172.21.15.89:0/1681715654 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f640faed0 0x7f0f640fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.819+0000 7f0f6a297700 1 -- 172.21.15.89:0/1681715654 >> 172.21.15.89:0/1681715654 conn(0x7f0f640f6950 msgr2=0x7f0f640f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.819+0000 7f0f6a297700 1 -- 172.21.15.89:0/1681715654 shutdown_connections 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.819+0000 7f0f6a297700 1 -- 172.21.15.89:0/1681715654 wait complete. 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.819+0000 7f0f6a297700 1 Processor -- start 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f6a297700 1 -- start start 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f6a297700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f640faed0 0x7f0f6410d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f6a297700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 0x7f0f6410db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f6a297700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6410e150 con 0x7f0f640faed0 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f6a297700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6410e9f0 con 0x7f0f640fdf60 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f5bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 0x7f0f6410db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f5bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 0x7f0f6410db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38678/0 (socket says 172.21.15.89:38678) 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f5bfff700 1 -- 172.21.15.89:0/2153124037 learned_addr learned my addr 172.21.15.89:0/2153124037 (peer_addr_for_me v2:172.21.15.89:0/0) 2023-12-19T18:44:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f63fff700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f640faed0 0x7f0f6410d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f5bfff700 1 -- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f640faed0 msgr2=0x7f0f6410d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f5bfff700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f640faed0 0x7f0f6410d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f5bfff700 1 -- 172.21.15.89:0/2153124037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c005040 con 0x7f0f640fdf60 2023-12-19T18:44:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f5bfff700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 0x7f0f6410db50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00b3b0 tx=0x7f0f4c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f617fa700 1 -- 172.21.15.89:0/2153124037 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f4c00e040 con 0x7f0f640fdf60 2023-12-19T18:44:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f617fa700 1 -- 172.21.15.89:0/2153124037 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f4c013420 con 0x7f0f640fdf60 2023-12-19T18:44:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6410ec70 con 0x7f0f640fdf60 2023-12-19T18:44:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f617fa700 1 -- 172.21.15.89:0/2153124037 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f4c01c650 con 0x7f0f640fdf60 2023-12-19T18:44:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.820+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6410f1b0 con 0x7f0f640fdf60 2023-12-19T18:44:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.821+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f641075f0 con 0x7f0f640fdf60 2023-12-19T18:44:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.822+0000 7f0f617fa700 1 -- 172.21.15.89:0/2153124037 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0f4c013590 con 0x7f0f640fdf60 2023-12-19T18:44:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.822+0000 7f0f617fa700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f50068f60 0x7f0f5006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.822+0000 7f0f617fa700 1 -- 172.21.15.89:0/2153124037 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0f4c087e10 con 0x7f0f640fdf60 2023-12-19T18:44:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.822+0000 7f0f63fff700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f50068f60 0x7f0f5006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.823+0000 7f0f63fff700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f50068f60 0x7f0f5006b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0f640650d0 tx=0x7f0f5400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:41.827 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:41.827+0000 7f0f617fa700 1 -- 172.21.15.89:0/2153124037 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0f4c058160 con 0x7f0f640fdf60 2023-12-19T18:44:41.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:41 smithi059 ceph-mon[80661]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:42.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:41 smithi089 ceph-mon[88459]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:42.089 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.088+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f0f6404bb80 con 0x7f0f640fdf60 2023-12-19T18:44:42.096 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.095+0000 7f0f617fa700 1 -- 172.21.15.89:0/2153124037 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0f4c057cf0 con 0x7f0f640fdf60 2023-12-19T18:44:42.096 INFO:teuthology.orchestra.run.smithi089.stdout:[client.1] 2023-12-19T18:44:42.096 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCa5IFlpZVmBRAAmK8gEKSCXEm1aXG2g/wT4w== 2023-12-19T18:44:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f50068f60 msgr2=0x7f0f5006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f50068f60 0x7f0f5006b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0f640650d0 tx=0x7f0f5400b040 comp rx=0 tx=0).stop 2023-12-19T18:44:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 msgr2=0x7f0f6410db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 0x7f0f6410db50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00b3b0 tx=0x7f0f4c009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 shutdown_connections 2023-12-19T18:44:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f50068f60 0x7f0f5006b410 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f640faed0 0x7f0f6410d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 --2- 172.21.15.89:0/2153124037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f640fdf60 0x7f0f6410db50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:42.098 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 >> 172.21.15.89:0/2153124037 conn(0x7f0f640f6950 msgr2=0x7f0f640f8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:42.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 shutdown_connections 2023-12-19T18:44:42.099 INFO:teuthology.orchestra.run.smithi089.stderr:2023-12-19T18:44:42.097+0000 7f0f6a297700 1 -- 172.21.15.89:0/2153124037 wait complete. 2023-12-19T18:44:42.613 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:44:42.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-12-19T18:44:42.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-19T18:44:42.656 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-19T18:44:42.656 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-12-19T18:44:42.656 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph mgr dump --format=json 2023-12-19T18:44:42.933 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:42.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:42 smithi059 ceph-mon[80661]: from='client.? 172.21.15.89:0/2153124037' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T18:44:42.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:42 smithi059 ceph-mon[80661]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T18:44:42.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:42 smithi059 ceph-mon[80661]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-19T18:44:42.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:42 smithi059 ceph-mon[80661]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:43.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:42 smithi089 ceph-mon[88459]: from='client.? 172.21.15.89:0/2153124037' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T18:44:43.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:42 smithi089 ceph-mon[88459]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T18:44:43.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:42 smithi089 ceph-mon[88459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-19T18:44:43.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:42 smithi089 ceph-mon[88459]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:44.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.470+0000 7fbfa3db4700 1 -- 172.21.15.59:0/534826917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 msgr2=0x7fbf9c0fbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.470+0000 7fbfa3db4700 1 --2- 172.21.15.59:0/534826917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 0x7fbf9c0fbf70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c004940 tx=0x7fbf8c009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.470+0000 7fbfa3db4700 1 -- 172.21.15.59:0/534826917 shutdown_connections 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.470+0000 7fbfa3db4700 1 --2- 172.21.15.59:0/534826917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 0x7fbf9c0fbf70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.470+0000 7fbfa3db4700 1 --2- 172.21.15.59:0/534826917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf9c101b60 0x7fbf9c101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.470+0000 7fbfa3db4700 1 -- 172.21.15.59:0/534826917 >> 172.21.15.59:0/534826917 conn(0x7fbf9c0f7680 msgr2=0x7fbf9c0f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.470+0000 7fbfa3db4700 1 -- 172.21.15.59:0/534826917 shutdown_connections 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.470+0000 7fbfa3db4700 1 -- 172.21.15.59:0/534826917 wait complete. 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa3db4700 1 Processor -- start 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa3db4700 1 -- start start 2023-12-19T18:44:44.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa3db4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 0x7fbf9c115870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa3db4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf9c101b60 0x7fbf9c115db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa3db4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c116400 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa3db4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c10f730 con 0x7fbf9c101b60 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa1b50700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 0x7fbf9c115870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa1b50700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 0x7fbf9c115870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52572/0 (socket says 172.21.15.59:52572) 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa1b50700 1 -- 172.21.15.59:0/3395162341 learned_addr learned my addr 172.21.15.59:0/3395162341 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa134f700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf9c101b60 0x7fbf9c115db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa1b50700 1 -- 172.21.15.59:0/3395162341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf9c101b60 msgr2=0x7fbf9c115db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa1b50700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf9c101b60 0x7fbf9c115db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:44.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa1b50700 1 -- 172.21.15.59:0/3395162341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c005040 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.471+0000 7fbfa134f700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf9c101b60 0x7fbf9c115db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T18:44:44.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.472+0000 7fbfa1b50700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 0x7fbf9c115870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800c8c0 tx=0x7fbf980044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:44.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.472+0000 7fbf927fc700 1 -- 172.21.15.59:0/3395162341 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf9800a980 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.472+0000 7fbf927fc700 1 -- 172.21.15.59:0/3395162341 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf9800ce60 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.472+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c10fa10 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.472+0000 7fbf927fc700 1 -- 172.21.15.59:0/3395162341 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf980113a0 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.472+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c10ff50 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.473+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c0fb280 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.473+0000 7fbf927fc700 1 -- 172.21.15.59:0/3395162341 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbf98012400 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.474+0000 7fbf927fc700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbf88068f70 0x7fbf8806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.474+0000 7fbf927fc700 1 -- 172.21.15.59:0/3395162341 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbf98086a70 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.474+0000 7fbfa134f700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbf88068f70 0x7fbf8806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:44.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.474+0000 7fbfa134f700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbf88068f70 0x7fbf8806b420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00b3b0 tx=0x7fbf8c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:44.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.478+0000 7fbf927fc700 1 -- 172.21.15.59:0/3395162341 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbf98056db0 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.733+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fbf9c110e50 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.736+0000 7fbf927fc700 1 -- 172.21.15.59:0/3395162341 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+166776 (secure 0 0 0) 0x7fbf98056940 con 0x7fbf9c0fbb00 2023-12-19T18:44:44.739 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:44:44.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.738+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbf88068f70 msgr2=0x7fbf8806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:44.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.738+0000 7fbfa3db4700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbf88068f70 0x7fbf8806b420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00b3b0 tx=0x7fbf8c009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:44.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.738+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 msgr2=0x7fbf9c115870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:44.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.738+0000 7fbfa3db4700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 0x7fbf9c115870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800c8c0 tx=0x7fbf980044c0 comp rx=0 tx=0).stop 2023-12-19T18:44:44.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.739+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 shutdown_connections 2023-12-19T18:44:44.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.739+0000 7fbfa3db4700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbf88068f70 0x7fbf8806b420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:44.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.739+0000 7fbfa3db4700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbf9c0fbb00 0x7fbf9c115870 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:44.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.739+0000 7fbfa3db4700 1 --2- 172.21.15.59:0/3395162341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf9c101b60 0x7fbf9c115db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:44.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.739+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 >> 172.21.15.59:0/3395162341 conn(0x7fbf9c0f7680 msgr2=0x7fbf9c0f86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:44.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.739+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 shutdown_connections 2023-12-19T18:44:44.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:44.739+0000 7fbfa3db4700 1 -- 172.21.15.59:0/3395162341 wait complete. 2023-12-19T18:44:45.379 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":19,"active_gid":14210,"active_name":"smithi059.ccbnob","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":3406589371},{"type":"v1","addr":"172.21.15.59:6801","nonce":3406589371}]},"active_addr":"172.21.15.59:6801/3406589371","active_change":"2023-12-19T18:41:08.487104+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi089.isaaav","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.59:8443/","prometheus":"http://172.21.15.59:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":3142679593}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":3785267407}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":2014868167}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.59:0","nonce":461921590}]}]} 2023-12-19T18:44:45.383 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-12-19T18:44:45.383 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-12-19T18:44:45.383 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-19T18:44:45.654 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:45.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:45 smithi059 ceph-mon[80661]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:45.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:45 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3395162341' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-19T18:44:46.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:45 smithi089 ceph-mon[88459]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:46.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:45 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3395162341' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-19T18:44:47.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 -- 172.21.15.59:0/1961778809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 msgr2=0x7f7e88101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 --2- 172.21.15.59:0/1961778809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 0x7f7e88101ed0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7e84004940 tx=0x7f7e84009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 -- 172.21.15.59:0/1961778809 shutdown_connections 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 --2- 172.21.15.59:0/1961778809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e880fbb40 0x7f7e880fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 --2- 172.21.15.59:0/1961778809 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 0x7f7e88101ed0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 -- 172.21.15.59:0/1961778809 >> 172.21.15.59:0/1961778809 conn(0x7f7e880f76c0 msgr2=0x7f7e880f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 -- 172.21.15.59:0/1961778809 shutdown_connections 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 -- 172.21.15.59:0/1961778809 wait complete. 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 Processor -- start 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.315+0000 7f7e8f888700 1 -- start start 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8f888700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e880fbb40 0x7f7e8810e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8f888700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 0x7f7e8810ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8f888700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8810f170 con 0x7f7e88101b00 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8f888700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8810f2b0 con 0x7f7e880fbb40 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8ce23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 0x7f7e8810ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8ce23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 0x7f7e8810ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52590/0 (socket says 172.21.15.59:52590) 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8ce23700 1 -- 172.21.15.59:0/1681517470 learned_addr learned my addr 172.21.15.59:0/1681517470 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8d624700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e880fbb40 0x7f7e8810e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8ce23700 1 -- 172.21.15.59:0/1681517470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e880fbb40 msgr2=0x7f7e8810e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8ce23700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e880fbb40 0x7f7e8810e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8ce23700 1 -- 172.21.15.59:0/1681517470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84005040 con 0x7f7e88101b00 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e8ce23700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 0x7f7e8810ebe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800eb70 tx=0x7f7e7800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.316+0000 7f7e7dffb700 1 -- 172.21.15.59:0/1681517470 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e78007cc0 con 0x7f7e88101b00 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.317+0000 7f7e7dffb700 1 -- 172.21.15.59:0/1681517470 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e7800f040 con 0x7f7e88101b00 2023-12-19T18:44:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.317+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e881175b0 con 0x7f7e88101b00 2023-12-19T18:44:47.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.317+0000 7f7e7dffb700 1 -- 172.21.15.59:0/1681517470 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e780095f0 con 0x7f7e88101b00 2023-12-19T18:44:47.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.317+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e88117ac0 con 0x7f7e88101b00 2023-12-19T18:44:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.318+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e88103fb0 con 0x7f7e88101b00 2023-12-19T18:44:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.318+0000 7f7e7dffb700 1 -- 172.21.15.59:0/1681517470 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7e78009750 con 0x7f7e88101b00 2023-12-19T18:44:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.319+0000 7f7e7dffb700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e74068f60 0x7f7e7406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.319+0000 7f7e7dffb700 1 -- 172.21.15.59:0/1681517470 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7e78086690 con 0x7f7e88101b00 2023-12-19T18:44:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.319+0000 7f7e8d624700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e74068f60 0x7f7e7406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.319+0000 7f7e8d624700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e74068f60 0x7f7e7406b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400b0d0 tx=0x7f7e84009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:47.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.323+0000 7f7e7dffb700 1 -- 172.21.15.59:0/1681517470 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7e78052e20 con 0x7f7e88101b00 2023-12-19T18:44:47.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.520+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e880fb2c0 con 0x7f7e88101b00 2023-12-19T18:44:47.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.521+0000 7f7e7dffb700 1 -- 172.21.15.59:0/1681517470 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11030 (secure 0 0 0) 0x7f7e78056440 con 0x7f7e88101b00 2023-12-19T18:44:47.522 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:44:47.522 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":41,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","created":"2023-12-19T18:38:57.167145+0000","modified":"2023-12-19T18:44:31.038588+0000","last_up_change":"2023-12-19T18:44:30.032151+0000","last_in_change":"2023-12-19T18:44:11.480055+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T18:43:21.014480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a33e1310-4169-41e4-bd1f-748898ef9289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6803","nonce":177591080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6805","nonce":177591080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6809","nonce":177591080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6807","nonce":177591080}]},"public_addr":"172.21.15.59:6803/177591080","cluster_addr":"172.21.15.59:6805/177591080","heartbeat_back_addr":"172.21.15.59:6809/177591080","heartbeat_front_addr":"172.21.15.59:6807/177591080","state":["exists","up"]},{"osd":1,"uuid":"73a2cb1c-90fe-4d56-adfd-78a688c6b2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6811","nonce":1239732570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6813","nonce":1239732570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6817","nonce":1239732570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6815","nonce":1239732570}]},"public_addr":"172.21.15.59:6811/1239732570","cluster_addr":"172.21.15.59:6813/1239732570","heartbeat_back_addr":"172.21.15.59:6817/1239732570","heartbeat_front_addr":"172.21.15.59:6815/1239732570","state":["exists","up"]},{"osd":2,"uuid":"0003bcc2-1e16-44d8-a031-8e26fa6c7467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6819","nonce":240742436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6821","nonce":240742436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6825","nonce":240742436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6823","nonce":240742436}]},"public_addr":"172.21.15.59:6819/240742436","cluster_addr":"172.21.15.59:6821/240742436","heartbeat_back_addr":"172.21.15.59:6825/240742436","heartbeat_front_addr":"172.21.15.59:6823/240742436","state":["exists","up"]},{"osd":3,"uuid":"08b40954-a1b3-4754-8494-e3a5423406c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6801","nonce":2655128462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6803","nonce":2655128462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6807","nonce":2655128462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6805","nonce":2655128462}]},"public_addr":"172.21.15.89:6801/2655128462","cluster_addr":"172.21.15.89:6803/2655128462","heartbeat_back_addr":"172.21.15.89:6807/2655128462","heartbeat_front_addr":"172.21.15.89:6805/2655128462","state":["exists","up"]},{"osd":4,"uuid":"152f485f-9a72-40e5-9a4d-535e4caf73ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6809","nonce":2501804384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6811","nonce":2501804384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6815","nonce":2501804384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6813","nonce":2501804384}]},"public_addr":"172.21.15.89:6809/2501804384","cluster_addr":"172.21.15.89:6811/2501804384","heartbeat_back_addr":"172.21.15.89:6815/2501804384","heartbeat_front_addr":"172.21.15.89:6813/2501804384","state":["exists","up"]},{"osd":5,"uuid":"a92c1e7d-9b7c-4594-b8db-7f812da2afc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6817","nonce":1872882017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6819","nonce":1872882017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6823","nonce":1872882017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6821","nonce":1872882017}]},"public_addr":"172.21.15.89:6817/1872882017","cluster_addr":"172.21.15.89:6819/1872882017","heartbeat_back_addr":"172.21.15.89:6823/1872882017","heartbeat_front_addr":"172.21.15.89:6821/1872882017","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:42:23.049328+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:42:50.671474+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:43:17.816383+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:43:40.185718+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:44:04.182008+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:44:28.308577+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/3325232385":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/3905471834":"2023-12-20T18:39:58.643904+0000","172.21.15.59:6801/6145510":"2023-12-20T18:41:08.486998+0000","172.21.15.59:6801/2067989811":"2023-12-20T18:39:58.643904+0000","172.21.15.59:6800/1973265269":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/3575411648":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/1003305302":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/3060878312":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/1665894963":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/990606911":"2023-12-20T18:39:27.397002+0000","172.21.15.59:6801/1973265269":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/1707232246":"2023-12-20T18:39:27.397002+0000","172.21.15.59:6800/6145510":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/4127123819":"2023-12-20T18:41:08.486998+0000","172.21.15.59:6800/2067989811":"2023-12-20T18:39:58.643904+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T18:44:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e74068f60 msgr2=0x7f7e7406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e74068f60 0x7f7e7406b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400b0d0 tx=0x7f7e84009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 msgr2=0x7f7e8810ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 0x7f7e8810ebe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800eb70 tx=0x7f7e7800b040 comp rx=0 tx=0).stop 2023-12-19T18:44:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 shutdown_connections 2023-12-19T18:44:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e880fbb40 0x7f7e8810e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e74068f60 0x7f7e7406b410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 --2- 172.21.15.59:0/1681517470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e88101b00 0x7f7e8810ebe0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:47.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 >> 172.21.15.59:0/1681517470 conn(0x7f7e880f76c0 msgr2=0x7f7e880f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:47.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 shutdown_connections 2023-12-19T18:44:47.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:47.523+0000 7f7e8f888700 1 -- 172.21.15.59:0/1681517470 wait complete. 2023-12-19T18:44:47.842 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:47 smithi059 ceph-mon[80661]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:47.842 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:47 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1681517470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T18:44:47.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:47 smithi089 ceph-mon[88459]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:47.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:47 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1681517470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T18:44:48.177 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-12-19T18:44:48.177 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-19T18:44:48.444 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:49.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:49 smithi089 ceph-mon[88459]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:49.841 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:49 smithi059 ceph-mon[80661]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.127+0000 7ff5f7833700 1 -- 172.21.15.59:0/541966281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f00fb1e0 msgr2=0x7ff5f00fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.127+0000 7ff5f7833700 1 --2- 172.21.15.59:0/541966281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f00fb1e0 0x7ff5f00fb630 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0004940 tx=0x7ff5e0009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:50.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.127+0000 7ff5f7833700 1 -- 172.21.15.59:0/541966281 shutdown_connections 2023-12-19T18:44:50.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.127+0000 7ff5f7833700 1 --2- 172.21.15.59:0/541966281 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f00fb1e0 0x7ff5f00fb630 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:50.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.127+0000 7ff5f7833700 1 --2- 172.21.15.59:0/541966281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f0102c70 0x7ff5f0103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:50.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.127+0000 7ff5f7833700 1 -- 172.21.15.59:0/541966281 >> 172.21.15.59:0/541966281 conn(0x7ff5f00711a0 msgr2=0x7ff5f00715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:50.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.127+0000 7ff5f7833700 1 -- 172.21.15.59:0/541966281 shutdown_connections 2023-12-19T18:44:50.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.127+0000 7ff5f7833700 1 -- 172.21.15.59:0/541966281 wait complete. 2023-12-19T18:44:50.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.128+0000 7ff5f7833700 1 Processor -- start 2023-12-19T18:44:50.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.128+0000 7ff5f7833700 1 -- start start 2023-12-19T18:44:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.128+0000 7ff5f7833700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f00fb1e0 0x7ff5f0113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.128+0000 7ff5f7833700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f0102c70 0x7ff5f0113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.128+0000 7ff5f7833700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0114200 con 0x7ff5f0102c70 2023-12-19T18:44:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.128+0000 7ff5f7833700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f010d4f0 con 0x7ff5f00fb1e0 2023-12-19T18:44:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.128+0000 7ff5f4dce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f0102c70 0x7ff5f0113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.128+0000 7ff5f4dce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f0102c70 0x7ff5f0113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52600/0 (socket says 172.21.15.59:52600) 2023-12-19T18:44:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.128+0000 7ff5f4dce700 1 -- 172.21.15.59:0/3903686536 learned_addr learned my addr 172.21.15.59:0/3903686536 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:50.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5f55cf700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f00fb1e0 0x7ff5f0113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5f4dce700 1 -- 172.21.15.59:0/3903686536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f00fb1e0 msgr2=0x7ff5f0113630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5f4dce700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f00fb1e0 0x7ff5f0113630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5f4dce700 1 -- 172.21.15.59:0/3903686536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e0005040 con 0x7ff5f0102c70 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5f55cf700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f00fb1e0 0x7ff5f0113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5f4dce700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f0102c70 0x7ff5f0113bb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000b3b0 tx=0x7ff5e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5e5ffb700 1 -- 172.21.15.59:0/3903686536 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5e000e040 con 0x7ff5f0102c70 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5e5ffb700 1 -- 172.21.15.59:0/3903686536 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5e000bc50 con 0x7ff5f0102c70 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5e5ffb700 1 -- 172.21.15.59:0/3903686536 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5e0013420 con 0x7ff5f0102c70 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f010d720 con 0x7ff5f0102c70 2023-12-19T18:44:50.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.129+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f010dc60 con 0x7ff5f0102c70 2023-12-19T18:44:50.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.130+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f0105050 con 0x7ff5f0102c70 2023-12-19T18:44:50.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.131+0000 7ff5e5ffb700 1 -- 172.21.15.59:0/3903686536 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff5e001c430 con 0x7ff5f0102c70 2023-12-19T18:44:50.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.131+0000 7ff5e5ffb700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff5dc068f60 0x7ff5dc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:50.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.131+0000 7ff5e5ffb700 1 -- 172.21.15.59:0/3903686536 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff5e0087830 con 0x7ff5f0102c70 2023-12-19T18:44:50.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.131+0000 7ff5f55cf700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff5dc068f60 0x7ff5dc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:50.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.132+0000 7ff5f55cf700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff5dc068f60 0x7ff5dc06b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0041d0 tx=0x7ff5ec009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:50.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.136+0000 7ff5e5ffb700 1 -- 172.21.15.59:0/3903686536 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff5e0057b80 con 0x7ff5f0102c70 2023-12-19T18:44:50.330 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.330+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5f0072cc0 con 0x7ff5f0102c70 2023-12-19T18:44:50.331 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.331+0000 7ff5e5ffb700 1 -- 172.21.15.59:0/3903686536 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11030 (secure 0 0 0) 0x7ff5e0057710 con 0x7ff5f0102c70 2023-12-19T18:44:50.331 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:44:50.331 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":41,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","created":"2023-12-19T18:38:57.167145+0000","modified":"2023-12-19T18:44:31.038588+0000","last_up_change":"2023-12-19T18:44:30.032151+0000","last_in_change":"2023-12-19T18:44:11.480055+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T18:43:21.014480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a33e1310-4169-41e4-bd1f-748898ef9289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6803","nonce":177591080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6805","nonce":177591080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6809","nonce":177591080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6807","nonce":177591080}]},"public_addr":"172.21.15.59:6803/177591080","cluster_addr":"172.21.15.59:6805/177591080","heartbeat_back_addr":"172.21.15.59:6809/177591080","heartbeat_front_addr":"172.21.15.59:6807/177591080","state":["exists","up"]},{"osd":1,"uuid":"73a2cb1c-90fe-4d56-adfd-78a688c6b2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6811","nonce":1239732570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6813","nonce":1239732570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6817","nonce":1239732570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6815","nonce":1239732570}]},"public_addr":"172.21.15.59:6811/1239732570","cluster_addr":"172.21.15.59:6813/1239732570","heartbeat_back_addr":"172.21.15.59:6817/1239732570","heartbeat_front_addr":"172.21.15.59:6815/1239732570","state":["exists","up"]},{"osd":2,"uuid":"0003bcc2-1e16-44d8-a031-8e26fa6c7467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6819","nonce":240742436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6821","nonce":240742436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6825","nonce":240742436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6823","nonce":240742436}]},"public_addr":"172.21.15.59:6819/240742436","cluster_addr":"172.21.15.59:6821/240742436","heartbeat_back_addr":"172.21.15.59:6825/240742436","heartbeat_front_addr":"172.21.15.59:6823/240742436","state":["exists","up"]},{"osd":3,"uuid":"08b40954-a1b3-4754-8494-e3a5423406c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6801","nonce":2655128462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6803","nonce":2655128462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6807","nonce":2655128462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6805","nonce":2655128462}]},"public_addr":"172.21.15.89:6801/2655128462","cluster_addr":"172.21.15.89:6803/2655128462","heartbeat_back_addr":"172.21.15.89:6807/2655128462","heartbeat_front_addr":"172.21.15.89:6805/2655128462","state":["exists","up"]},{"osd":4,"uuid":"152f485f-9a72-40e5-9a4d-535e4caf73ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6809","nonce":2501804384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6811","nonce":2501804384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6815","nonce":2501804384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6813","nonce":2501804384}]},"public_addr":"172.21.15.89:6809/2501804384","cluster_addr":"172.21.15.89:6811/2501804384","heartbeat_back_addr":"172.21.15.89:6815/2501804384","heartbeat_front_addr":"172.21.15.89:6813/2501804384","state":["exists","up"]},{"osd":5,"uuid":"a92c1e7d-9b7c-4594-b8db-7f812da2afc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6817","nonce":1872882017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6819","nonce":1872882017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6823","nonce":1872882017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6821","nonce":1872882017}]},"public_addr":"172.21.15.89:6817/1872882017","cluster_addr":"172.21.15.89:6819/1872882017","heartbeat_back_addr":"172.21.15.89:6823/1872882017","heartbeat_front_addr":"172.21.15.89:6821/1872882017","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:42:23.049328+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:42:50.671474+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:43:17.816383+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:43:40.185718+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:44:04.182008+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:44:28.308577+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:0/3325232385":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/3905471834":"2023-12-20T18:39:58.643904+0000","172.21.15.59:6801/6145510":"2023-12-20T18:41:08.486998+0000","172.21.15.59:6801/2067989811":"2023-12-20T18:39:58.643904+0000","172.21.15.59:6800/1973265269":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/3575411648":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/1003305302":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/3060878312":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/1665894963":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/990606911":"2023-12-20T18:39:27.397002+0000","172.21.15.59:6801/1973265269":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/1707232246":"2023-12-20T18:39:27.397002+0000","172.21.15.59:6800/6145510":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/4127123819":"2023-12-20T18:41:08.486998+0000","172.21.15.59:6800/2067989811":"2023-12-20T18:39:58.643904+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff5dc068f60 msgr2=0x7ff5dc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff5dc068f60 0x7ff5dc06b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0041d0 tx=0x7ff5ec009040 comp rx=0 tx=0).stop 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f0102c70 msgr2=0x7ff5f0113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f0102c70 0x7ff5f0113bb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000b3b0 tx=0x7ff5e0009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 shutdown_connections 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5f00fb1e0 0x7ff5f0113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff5dc068f60 0x7ff5dc06b410 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 --2- 172.21.15.59:0/3903686536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5f0102c70 0x7ff5f0113bb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 >> 172.21.15.59:0/3903686536 conn(0x7ff5f00711a0 msgr2=0x7ff5f00fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 shutdown_connections 2023-12-19T18:44:50.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:50.333+0000 7ff5f7833700 1 -- 172.21.15.59:0/3903686536 wait complete. 2023-12-19T18:44:50.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:50 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3903686536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T18:44:50.998 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2023-12-19T18:44:50.998 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2023-12-19T18:44:50.999 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2023-12-19T18:44:50.999 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2023-12-19T18:44:50.999 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2023-12-19T18:44:51.000 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2023-12-19T18:44:51.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:50 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3903686536' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T18:44:51.337 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:51.338 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:51.708 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:51 smithi059 ceph-mon[80661]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:52.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:51 smithi089 ceph-mon[88459]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:53.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:52 smithi059 ceph-mon[80661]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:53.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:52 smithi089 ceph-mon[88459]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:53.311 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:53.312 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:53.908 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:54.028 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:44:54.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:44:55.179 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:54 smithi059 ceph-mon[80661]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:55.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.197+0000 7f78d5193700 1 -- 172.21.15.59:0/1975290621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d00fbb00 msgr2=0x7f78d00fbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.197+0000 7f78d5193700 1 --2- 172.21.15.59:0/1975290621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d00fbb00 0x7f78d00fbf70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f78c0004940 tx=0x7f78c0009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:55.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.198+0000 7f78d5193700 1 -- 172.21.15.59:0/1975290621 shutdown_connections 2023-12-19T18:44:55.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.198+0000 7f78d5193700 1 --2- 172.21.15.59:0/1975290621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d00fbb00 0x7f78d00fbf70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.198+0000 7f78d5193700 1 --2- 172.21.15.59:0/1975290621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78d0101b60 0x7f78d0101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.198+0000 7f78d5193700 1 -- 172.21.15.59:0/1975290621 >> 172.21.15.59:0/1975290621 conn(0x7f78d00f7680 msgr2=0x7f78d00f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.198+0000 7f78d5193700 1 -- 172.21.15.59:0/1975290621 shutdown_connections 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.198+0000 7f78d5193700 1 -- 172.21.15.59:0/1975290621 wait complete. 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.198+0000 7f78d5193700 1 Processor -- start 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.198+0000 7f78d5193700 1 -- start start 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.198+0000 7f78d5193700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78d00fbb00 0x7f78d010c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78d5193700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d0101b60 0x7f78d010c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78d5193700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d010cd60 con 0x7f78d0101b60 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78d5193700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d010cea0 con 0x7f78d00fbb00 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78ce59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d0101b60 0x7f78d010c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:55.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78ce59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d0101b60 0x7f78d010c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34694/0 (socket says 172.21.15.59:34694) 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78ce59c700 1 -- 172.21.15.59:0/1809719487 learned_addr learned my addr 172.21.15.59:0/1809719487 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78ced9d700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78d00fbb00 0x7f78d010c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78ce59c700 1 -- 172.21.15.59:0/1809719487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78d00fbb00 msgr2=0x7f78d010c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78ce59c700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78d00fbb00 0x7f78d010c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78ce59c700 1 -- 172.21.15.59:0/1809719487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c0005040 con 0x7f78d0101b60 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78ced9d700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78d00fbb00 0x7f78d010c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78ce59c700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d0101b60 0x7f78d010c820 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78c000b3b0 tx=0x7f78c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78b77fe700 1 -- 172.21.15.59:0/1809719487 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78c000e040 con 0x7f78d0101b60 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d010d120 con 0x7f78d0101b60 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78b77fe700 1 -- 172.21.15.59:0/1809719487 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78c0004290 con 0x7f78d0101b60 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78b77fe700 1 -- 172.21.15.59:0/1809719487 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78c001c3f0 con 0x7f78d0101b60 2023-12-19T18:44:55.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.199+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d0119cd0 con 0x7f78d0101b60 2023-12-19T18:44:55.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.200+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78d005e380 con 0x7f78d0101b60 2023-12-19T18:44:55.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.201+0000 7f78b77fe700 1 -- 172.21.15.59:0/1809719487 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f78c000bc50 con 0x7f78d0101b60 2023-12-19T18:44:55.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.201+0000 7f78b77fe700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78b8068f60 0x7f78b806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:55.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.201+0000 7f78b77fe700 1 -- 172.21.15.59:0/1809719487 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f78c0088940 con 0x7f78d0101b60 2023-12-19T18:44:55.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.201+0000 7f78ced9d700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78b8068f60 0x7f78b806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:55.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.202+0000 7f78b77fe700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] conn(0x7f78b806ec30 0x7f78b8071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:55.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.202+0000 7f78b77fe700 1 -- 172.21.15.59:0/1809719487 --> [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78b80716f0 con 0x7f78b806ec30 2023-12-19T18:44:55.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.202+0000 7f78b77fe700 1 -- 172.21.15.59:0/1809719487 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78c0088d20 con 0x7f78d0101b60 2023-12-19T18:44:55.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.202+0000 7f78cf59e700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] conn(0x7f78b806ec30 0x7f78b8071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:55.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.202+0000 7f78ced9d700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78b8068f60 0x7f78b806b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009fd0 tx=0x7f78c4009ee0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:55.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.202+0000 7f78cf59e700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] conn(0x7f78b806ec30 0x7f78b8071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:55.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.203+0000 7f78b77fe700 1 -- 172.21.15.59:0/1809719487 <== osd.0 v2:172.21.15.59:6802/177591080 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f78b80716f0 con 0x7f78b806ec30 2023-12-19T18:44:55.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.218+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 --> [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f78d005e800 con 0x7f78b806ec30 2023-12-19T18:44:55.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.218+0000 7f78b77fe700 1 -- 172.21.15.59:0/1809719487 <== osd.0 v2:172.21.15.59:6802/177591080 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f78d005e800 con 0x7f78b806ec30 2023-12-19T18:44:55.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.218+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] conn(0x7f78b806ec30 msgr2=0x7f78b8071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.218+0000 7f78d5193700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] conn(0x7f78b806ec30 0x7f78b8071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.218+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78b8068f60 msgr2=0x7f78b806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.218+0000 7f78d5193700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78b8068f60 0x7f78b806b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009fd0 tx=0x7f78c4009ee0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.218+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d0101b60 msgr2=0x7f78d010c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.218+0000 7f78d5193700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d0101b60 0x7f78d010c820 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f78c000b3b0 tx=0x7f78c0009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.219+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 shutdown_connections 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.219+0000 7f78d5193700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6802/177591080,v1:172.21.15.59:6803/177591080] conn(0x7f78b806ec30 0x7f78b8071040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.219+0000 7f78d5193700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78d00fbb00 0x7f78d010c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.219+0000 7f78d5193700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f78b8068f60 0x7f78b806b410 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.219+0000 7f78d5193700 1 --2- 172.21.15.59:0/1809719487 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78d0101b60 0x7f78d010c820 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.219+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 >> 172.21.15.59:0/1809719487 conn(0x7f78d00f7680 msgr2=0x7f78d00f8350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.219+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 shutdown_connections 2023-12-19T18:44:55.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.219+0000 7f78d5193700 1 -- 172.21.15.59:0/1809719487 wait complete. 2023-12-19T18:44:55.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:54 smithi089 ceph-mon[88459]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 -- 172.21.15.59:0/4074887463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 msgr2=0x7f52c00fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 --2- 172.21.15.59:0/4074887463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 0x7f52c00fbfb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f52b4004940 tx=0x7f52b4009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 -- 172.21.15.59:0/4074887463 shutdown_connections 2023-12-19T18:44:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 --2- 172.21.15.59:0/4074887463 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 0x7f52c00fbfb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 --2- 172.21.15.59:0/4074887463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0101b00 0x7f52c0101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 -- 172.21.15.59:0/4074887463 >> 172.21.15.59:0/4074887463 conn(0x7f52c00719b0 msgr2=0x7f52c0071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 -- 172.21.15.59:0/4074887463 shutdown_connections 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 -- 172.21.15.59:0/4074887463 wait complete. 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 Processor -- start 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.740+0000 7f52c878c700 1 -- start start 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c878c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 0x7f52c0112640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c878c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0101b00 0x7f52c0112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c878c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c01131d0 con 0x7f52c00fbb40 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c878c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c010c4c0 con 0x7f52c0101b00 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c6528700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 0x7f52c0112640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c6528700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 0x7f52c0112640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34702/0 (socket says 172.21.15.59:34702) 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c6528700 1 -- 172.21.15.59:0/641950262 learned_addr learned my addr 172.21.15.59:0/641950262 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c5d27700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0101b00 0x7f52c0112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c6528700 1 -- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0101b00 msgr2=0x7f52c0112b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c6528700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0101b00 0x7f52c0112b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c6528700 1 -- 172.21.15.59:0/641950262 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b4005040 con 0x7f52c00fbb40 2023-12-19T18:44:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c6528700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 0x7f52c0112640 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f52b000cb10 tx=0x7f52b0007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52beffd700 1 -- 172.21.15.59:0/641950262 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52b0004200 con 0x7f52c00fbb40 2023-12-19T18:44:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52beffd700 1 -- 172.21.15.59:0/641950262 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52b000ac50 con 0x7f52c00fbb40 2023-12-19T18:44:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c010c7a0 con 0x7f52c00fbb40 2023-12-19T18:44:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.741+0000 7f52beffd700 1 -- 172.21.15.59:0/641950262 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52b0003ce0 con 0x7f52c00fbb40 2023-12-19T18:44:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.742+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c010cce0 con 0x7f52c00fbb40 2023-12-19T18:44:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.742+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f52c010d1b0 con 0x7f52c00fbb40 2023-12-19T18:44:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.743+0000 7f52beffd700 1 -- 172.21.15.59:0/641950262 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f52b000a7a0 con 0x7f52c00fbb40 2023-12-19T18:44:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.743+0000 7f52beffd700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f52a8068f60 0x7f52a806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.743+0000 7f52beffd700 1 -- 172.21.15.59:0/641950262 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f52b0086f70 con 0x7f52c00fbb40 2023-12-19T18:44:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.743+0000 7f52c5d27700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f52a8068f60 0x7f52a806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.744+0000 7f52beffd700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] conn(0x7f52a806ec30 0x7f52a8071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.744+0000 7f52beffd700 1 -- 172.21.15.59:0/641950262 --> [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52a80716f0 con 0x7f52a806ec30 2023-12-19T18:44:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.744+0000 7f52beffd700 1 -- 172.21.15.59:0/641950262 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52b00894c0 con 0x7f52c00fbb40 2023-12-19T18:44:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.744+0000 7f52c5d27700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f52a8068f60 0x7f52a806b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52c010dea0 tx=0x7f52b4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.744+0000 7f52c6d29700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] conn(0x7f52a806ec30 0x7f52a8071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.744+0000 7f52c6d29700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] conn(0x7f52a806ec30 0x7f52a8071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.745+0000 7f52beffd700 1 -- 172.21.15.59:0/641950262 <== osd.4 v2:172.21.15.89:6808/2501804384 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f52a80716f0 con 0x7f52a806ec30 2023-12-19T18:44:55.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.760+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 --> [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f52c0065630 con 0x7f52a806ec30 2023-12-19T18:44:55.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52beffd700 1 -- 172.21.15.59:0/641950262 <== osd.4 v2:172.21.15.89:6808/2501804384 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f52c0065630 con 0x7f52a806ec30 2023-12-19T18:44:55.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] conn(0x7f52a806ec30 msgr2=0x7f52a8071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] conn(0x7f52a806ec30 0x7f52a8071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f52a8068f60 msgr2=0x7f52a806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f52a8068f60 0x7f52a806b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52c010dea0 tx=0x7f52b4009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:55.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 msgr2=0x7f52c0112640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:55.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 0x7f52c0112640 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f52b000cb10 tx=0x7f52b0007f90 comp rx=0 tx=0).stop 2023-12-19T18:44:55.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 shutdown_connections 2023-12-19T18:44:55.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:6808/2501804384,v1:172.21.15.89:6809/2501804384] conn(0x7f52a806ec30 0x7f52a8071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f52a8068f60 0x7f52a806b410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f52c00fbb40 0x7f52c0112640 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 --2- 172.21.15.59:0/641950262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c0101b00 0x7f52c0112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 >> 172.21.15.59:0/641950262 conn(0x7f52c00719b0 msgr2=0x7f52c00fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 shutdown_connections 2023-12-19T18:44:55.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:55.761+0000 7f52c878c700 1 -- 172.21.15.59:0/641950262 wait complete. 2023-12-19T18:44:56.302 INFO:teuthology.orchestra.run.smithi059.stdout:38654705697 2023-12-19T18:44:56.303 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.0 2023-12-19T18:44:56.720 INFO:teuthology.orchestra.run.smithi059.stdout:150323855372 2023-12-19T18:44:56.721 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.4 2023-12-19T18:44:57.886 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:57 smithi089 ceph-mon[88459]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:57.937 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:57 smithi059 ceph-mon[80661]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.027+0000 7f9795108700 1 -- 172.21.15.59:0/3610972489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790102c20 msgr2=0x7f9790102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.027+0000 7f9795108700 1 --2- 172.21.15.59:0/3610972489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790102c20 0x7f9790102ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9780004940 tx=0x7f9780009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.028+0000 7f9795108700 1 -- 172.21.15.59:0/3610972489 shutdown_connections 2023-12-19T18:44:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.028+0000 7f9795108700 1 --2- 172.21.15.59:0/3610972489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9790065390 0x7f9790065800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.028+0000 7f9795108700 1 --2- 172.21.15.59:0/3610972489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790102c20 0x7f9790102ff0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.028+0000 7f9795108700 1 -- 172.21.15.59:0/3610972489 >> 172.21.15.59:0/3610972489 conn(0x7f97900719b0 msgr2=0x7f9790071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.028+0000 7f9795108700 1 -- 172.21.15.59:0/3610972489 shutdown_connections 2023-12-19T18:44:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.028+0000 7f9795108700 1 -- 172.21.15.59:0/3610972489 wait complete. 2023-12-19T18:44:59.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f9795108700 1 Processor -- start 2023-12-19T18:44:59.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f9795108700 1 -- start start 2023-12-19T18:44:59.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f9795108700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790065390 0x7f979006a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:59.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f9795108700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9790102c20 0x7f979006ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:59.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f9795108700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979006fa90 con 0x7f9790065390 2023-12-19T18:44:59.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f9795108700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979006fbd0 con 0x7f9790102c20 2023-12-19T18:44:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f978ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790065390 0x7f979006a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f978ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790065390 0x7f979006a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34728/0 (socket says 172.21.15.59:34728) 2023-12-19T18:44:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f978ed9d700 1 -- 172.21.15.59:0/4221576590 learned_addr learned my addr 172.21.15.59:0/4221576590 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.029+0000 7f978e59c700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9790102c20 0x7f979006ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.030+0000 7f978e59c700 1 -- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790065390 msgr2=0x7f979006a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.030+0000 7f978e59c700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790065390 0x7f979006a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.030+0000 7f978e59c700 1 -- 172.21.15.59:0/4221576590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9780005040 con 0x7f9790102c20 2023-12-19T18:44:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.030+0000 7f978ed9d700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790065390 0x7f979006a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:44:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.030+0000 7f978e59c700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9790102c20 0x7f979006ad70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9784002820 tx=0x7f9784004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.031+0000 7f97777fe700 1 -- 172.21.15.59:0/4221576590 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9784007cc0 con 0x7f9790102c20 2023-12-19T18:44:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.031+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979006b310 con 0x7f9790102c20 2023-12-19T18:44:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.031+0000 7f97777fe700 1 -- 172.21.15.59:0/4221576590 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f978400f040 con 0x7f9790102c20 2023-12-19T18:44:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.031+0000 7f97777fe700 1 -- 172.21.15.59:0/4221576590 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9784007cc0 con 0x7f9790102c20 2023-12-19T18:44:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.031+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979006b690 con 0x7f9790102c20 2023-12-19T18:44:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.032+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f977c000fd0 con 0x7f9790102c20 2023-12-19T18:44:59.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.033+0000 7f97777fe700 1 -- 172.21.15.59:0/4221576590 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9784013730 con 0x7f9790102c20 2023-12-19T18:44:59.034 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:44:59.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.033+0000 7f97777fe700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9778069020 0x7f977806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:59.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.033+0000 7f97777fe700 1 -- 172.21.15.59:0/4221576590 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f97840862a0 con 0x7f9790102c20 2023-12-19T18:44:59.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.034+0000 7f978ed9d700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9778069020 0x7f977806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.034+0000 7f97777fe700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] conn(0x7f977806ecf0 0x7f9778071100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.034+0000 7f97777fe700 1 -- 172.21.15.59:0/4221576590 --> [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97780717b0 con 0x7f977806ecf0 2023-12-19T18:44:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.034+0000 7f97777fe700 1 -- 172.21.15.59:0/4221576590 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9784007cc0 con 0x7f9790102c20 2023-12-19T18:44:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.034+0000 7f978f59e700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] conn(0x7f977806ecf0 0x7f9778071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.034+0000 7f978ed9d700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9778069020 0x7f977806b4d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9780009fd0 tx=0x7f9780009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.034+0000 7f978f59e700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] conn(0x7f977806ecf0 0x7f9778071100 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.035+0000 7f97777fe700 1 -- 172.21.15.59:0/4221576590 <== osd.3 v2:172.21.15.89:6800/2655128462 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f97780717b0 con 0x7f977806ecf0 2023-12-19T18:44:59.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.052+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 --> [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f977c002d30 con 0x7f977806ecf0 2023-12-19T18:44:59.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.053+0000 7f97777fe700 1 -- 172.21.15.59:0/4221576590 <== osd.3 v2:172.21.15.89:6800/2655128462 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f977c002d30 con 0x7f977806ecf0 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.053+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] conn(0x7f977806ecf0 msgr2=0x7f9778071100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.053+0000 7f9795108700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] conn(0x7f977806ecf0 0x7f9778071100 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9778069020 msgr2=0x7f977806b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9778069020 0x7f977806b4d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9780009fd0 tx=0x7f9780009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9790102c20 msgr2=0x7f979006ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9790102c20 0x7f979006ad70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9784002820 tx=0x7f9784004570 comp rx=0 tx=0).stop 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 shutdown_connections 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9778069020 0x7f977806b4d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9790065390 0x7f979006a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:6800/2655128462,v1:172.21.15.89:6801/2655128462] conn(0x7f977806ecf0 0x7f9778071100 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 --2- 172.21.15.59:0/4221576590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9790102c20 0x7f979006ad70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 >> 172.21.15.59:0/4221576590 conn(0x7f97900719b0 msgr2=0x7f97900fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 shutdown_connections 2023-12-19T18:44:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.054+0000 7f9795108700 1 -- 172.21.15.59:0/4221576590 wait complete. 2023-12-19T18:44:59.755 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:44:59 smithi059 ceph-mon[80661]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:44:59.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.785+0000 7efff0c14700 1 -- 172.21.15.59:0/2251456002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec1057a0 msgr2=0x7effec0fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.785+0000 7efff0c14700 1 --2- 172.21.15.59:0/2251456002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec1057a0 0x7effec0fee10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7effdc004940 tx=0x7effdc009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:59.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.785+0000 7efff0c14700 1 -- 172.21.15.59:0/2251456002 shutdown_connections 2023-12-19T18:44:59.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.785+0000 7efff0c14700 1 --2- 172.21.15.59:0/2251456002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec1057a0 0x7effec0fee10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.785+0000 7efff0c14700 1 --2- 172.21.15.59:0/2251456002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effec104e00 0x7effec1051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.785+0000 7efff0c14700 1 -- 172.21.15.59:0/2251456002 >> 172.21.15.59:0/2251456002 conn(0x7effec0fa990 msgr2=0x7effec0fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:59.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.785+0000 7efff0c14700 1 -- 172.21.15.59:0/2251456002 shutdown_connections 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.785+0000 7efff0c14700 1 -- 172.21.15.59:0/2251456002 wait complete. 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7efff0c14700 1 Processor -- start 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7efff0c14700 1 -- start start 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7efff0c14700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec104e00 0x7effec115820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7efff0c14700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effec1057a0 0x7effec115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7efff0c14700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec1163b0 con 0x7effec104e00 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7efff0c14700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec10f6e0 con 0x7effec1057a0 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7effea59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec104e00 0x7effec115820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7effea59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec104e00 0x7effec115820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34740/0 (socket says 172.21.15.59:34740) 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7effea59c700 1 -- 172.21.15.59:0/2669532963 learned_addr learned my addr 172.21.15.59:0/2669532963 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7effe9d9b700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effec1057a0 0x7effec115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7effea59c700 1 -- 172.21.15.59:0/2669532963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effec1057a0 msgr2=0x7effec115d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7effea59c700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effec1057a0 0x7effec115d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7effea59c700 1 -- 172.21.15.59:0/2669532963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc005040 con 0x7effec104e00 2023-12-19T18:44:59.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.786+0000 7effea59c700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec104e00 0x7effec115820 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7effe000eb40 tx=0x7effe000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.787+0000 7effdaffd700 1 -- 172.21.15.59:0/2669532963 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe0007cc0 con 0x7effec104e00 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.787+0000 7effdaffd700 1 -- 172.21.15.59:0/2669532963 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effe000f040 con 0x7effec104e00 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.787+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effec10f960 con 0x7effec104e00 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.787+0000 7effdaffd700 1 -- 172.21.15.59:0/2669532963 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effe000c990 con 0x7effec104e00 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.787+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effec10fe70 con 0x7effec104e00 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.788+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7effec110370 con 0x7effec104e00 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.788+0000 7effdaffd700 1 -- 172.21.15.59:0/2669532963 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7effe000caf0 con 0x7effec104e00 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.789+0000 7effdaffd700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7effd4068f60 0x7effd406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.789+0000 7effdaffd700 1 -- 172.21.15.59:0/2669532963 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7effe0086590 con 0x7effec104e00 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.789+0000 7effe9d9b700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7effd4068f60 0x7effd406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.789+0000 7effdaffd700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] conn(0x7effd406ec30 0x7effd4071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:44:59.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.789+0000 7effdaffd700 1 -- 172.21.15.59:0/2669532963 --> [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7effd40716f0 con 0x7effd406ec30 2023-12-19T18:44:59.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.789+0000 7effdaffd700 1 -- 172.21.15.59:0/2669532963 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7effe0088b80 con 0x7effec104e00 2023-12-19T18:44:59.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.789+0000 7effead9d700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] conn(0x7effd406ec30 0x7effd4071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:44:59.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.789+0000 7effe9d9b700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7effd4068f60 0x7effd406b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effdc000c00 tx=0x7effdc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:59.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.789+0000 7effead9d700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] conn(0x7effd406ec30 0x7effd4071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:44:59.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.790+0000 7effdaffd700 1 -- 172.21.15.59:0/2669532963 <== osd.1 v2:172.21.15.59:6810/1239732570 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7effd40716f0 con 0x7effd406ec30 2023-12-19T18:44:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.805+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 --> [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7effec110530 con 0x7effd406ec30 2023-12-19T18:44:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7effdaffd700 1 -- 172.21.15.59:0/2669532963 <== osd.1 v2:172.21.15.59:6810/1239732570 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7effec110530 con 0x7effd406ec30 2023-12-19T18:44:59.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] conn(0x7effd406ec30 msgr2=0x7effd4071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] conn(0x7effd406ec30 0x7effd4071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7effd4068f60 msgr2=0x7effd406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7effd4068f60 0x7effd406b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effdc000c00 tx=0x7effdc009e30 comp rx=0 tx=0).stop 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec104e00 msgr2=0x7effec115820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec104e00 0x7effec115820 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7effe000eb40 tx=0x7effe000b040 comp rx=0 tx=0).stop 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 shutdown_connections 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6810/1239732570,v1:172.21.15.59:6811/1239732570] conn(0x7effd406ec30 0x7effd4071040 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7effd4068f60 0x7effd406b410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7effec104e00 0x7effec115820 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 --2- 172.21.15.59:0/2669532963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effec1057a0 0x7effec115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 >> 172.21.15.59:0/2669532963 conn(0x7effec0fa990 msgr2=0x7effec0fc490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 shutdown_connections 2023-12-19T18:44:59.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:44:59.806+0000 7efff0c14700 1 -- 172.21.15.59:0/2669532963 wait complete. 2023-12-19T18:45:00.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:44:59 smithi089 ceph-mon[88459]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:00.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.346+0000 7f5af9115700 1 -- 172.21.15.59:0/1732327689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af40fbb40 msgr2=0x7f5af40fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:00.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.346+0000 7f5af9115700 1 --2- 172.21.15.59:0/1732327689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af40fbb40 0x7f5af40fbfb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5adc004940 tx=0x7f5adc009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:00.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.346+0000 7f5af9115700 1 -- 172.21.15.59:0/1732327689 shutdown_connections 2023-12-19T18:45:00.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.346+0000 7f5af9115700 1 --2- 172.21.15.59:0/1732327689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af40fbb40 0x7f5af40fbfb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:00.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.346+0000 7f5af9115700 1 --2- 172.21.15.59:0/1732327689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af4101b00 0x7f5af4101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:00.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.346+0000 7f5af9115700 1 -- 172.21.15.59:0/1732327689 >> 172.21.15.59:0/1732327689 conn(0x7f5af40719b0 msgr2=0x7f5af4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.346+0000 7f5af9115700 1 -- 172.21.15.59:0/1732327689 shutdown_connections 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.346+0000 7f5af9115700 1 -- 172.21.15.59:0/1732327689 wait complete. 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af9115700 1 Processor -- start 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af9115700 1 -- start start 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af9115700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af40fbb40 0x7f5af4112640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af9115700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af4101b00 0x7f5af4112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af9115700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af41131d0 con 0x7f5af4101b00 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af9115700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af410c4c0 con 0x7f5af40fbb40 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af4101b00 0x7f5af4112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:00.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af4101b00 0x7f5af4112b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34766/0 (socket says 172.21.15.59:34766) 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af259c700 1 -- 172.21.15.59:0/178266225 learned_addr learned my addr 172.21.15.59:0/178266225 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af2d9d700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af40fbb40 0x7f5af4112640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af259c700 1 -- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af40fbb40 msgr2=0x7f5af4112640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af259c700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af40fbb40 0x7f5af4112640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af259c700 1 -- 172.21.15.59:0/178266225 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc005040 con 0x7f5af4101b00 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.347+0000 7f5af259c700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af4101b00 0x7f5af4112b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00b3b0 tx=0x7f5adc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.348+0000 7f5aeb7fe700 1 -- 172.21.15.59:0/178266225 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5adc00e040 con 0x7f5af4101b00 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.348+0000 7f5aeb7fe700 1 -- 172.21.15.59:0/178266225 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5adc01c430 con 0x7f5af4101b00 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.348+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af410c740 con 0x7f5af4101b00 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.348+0000 7f5aeb7fe700 1 -- 172.21.15.59:0/178266225 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5adc013640 con 0x7f5af4101b00 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.348+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af410cc80 con 0x7f5af4101b00 2023-12-19T18:45:00.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.348+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5af410d170 con 0x7f5af4101b00 2023-12-19T18:45:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.349+0000 7f5aeb7fe700 1 -- 172.21.15.59:0/178266225 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f5adc01c5a0 con 0x7f5af4101b00 2023-12-19T18:45:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.349+0000 7f5aeb7fe700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5ae0068f60 0x7f5ae006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.350+0000 7f5aeb7fe700 1 -- 172.21.15.59:0/178266225 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5adc087450 con 0x7f5af4101b00 2023-12-19T18:45:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.350+0000 7f5af2d9d700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5ae0068f60 0x7f5ae006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.350+0000 7f5aeb7fe700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] conn(0x7f5ae006ec30 0x7f5ae0071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.350+0000 7f5aeb7fe700 1 -- 172.21.15.59:0/178266225 --> [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ae00716f0 con 0x7f5ae006ec30 2023-12-19T18:45:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.350+0000 7f5aeb7fe700 1 -- 172.21.15.59:0/178266225 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5adc089a60 con 0x7f5af4101b00 2023-12-19T18:45:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.350+0000 7f5af2d9d700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5ae0068f60 0x7f5ae006b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4007a40 tx=0x7f5ae4009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:00.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.350+0000 7f5af359e700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] conn(0x7f5ae006ec30 0x7f5ae0071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:00.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.350+0000 7f5af359e700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] conn(0x7f5ae006ec30 0x7f5ae0071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:00.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.351+0000 7f5aeb7fe700 1 -- 172.21.15.59:0/178266225 <== osd.5 v2:172.21.15.89:6816/1872882017 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f5ae00716f0 con 0x7f5ae006ec30 2023-12-19T18:45:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.366+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 --> [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5af405e2c0 con 0x7f5ae006ec30 2023-12-19T18:45:00.367 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.367+0000 7f5aeb7fe700 1 -- 172.21.15.59:0/178266225 <== osd.5 v2:172.21.15.89:6816/1872882017 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5af405e2c0 con 0x7f5ae006ec30 2023-12-19T18:45:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.367+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] conn(0x7f5ae006ec30 msgr2=0x7f5ae0071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.367+0000 7f5af9115700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] conn(0x7f5ae006ec30 0x7f5ae0071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.367+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5ae0068f60 msgr2=0x7f5ae006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.367+0000 7f5af9115700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5ae0068f60 0x7f5ae006b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4007a40 tx=0x7f5ae4009da0 comp rx=0 tx=0).stop 2023-12-19T18:45:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.367+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af4101b00 msgr2=0x7f5af4112b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.367+0000 7f5af9115700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af4101b00 0x7f5af4112b80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00b3b0 tx=0x7f5adc009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.367+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 shutdown_connections 2023-12-19T18:45:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.367+0000 7f5af9115700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:6816/1872882017,v1:172.21.15.89:6817/1872882017] conn(0x7f5ae006ec30 0x7f5ae0071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:00.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.368+0000 7f5af9115700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5af40fbb40 0x7f5af4112640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.368+0000 7f5af9115700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5ae0068f60 0x7f5ae006b410 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.368+0000 7f5af9115700 1 --2- 172.21.15.59:0/178266225 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af4101b00 0x7f5af4112b80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.368+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 >> 172.21.15.59:0/178266225 conn(0x7f5af40719b0 msgr2=0x7f5af40fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.368+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 shutdown_connections 2023-12-19T18:45:00.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:00.368+0000 7f5af9115700 1 -- 172.21.15.59:0/178266225 wait complete. 2023-12-19T18:45:01.070 INFO:teuthology.orchestra.run.smithi059.stdout:60129542172 2023-12-19T18:45:01.070 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.1 2023-12-19T18:45:01.659 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:01 smithi059 ceph-mon[80661]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:01.683 INFO:teuthology.orchestra.run.smithi059.stdout:171798691848 2023-12-19T18:45:01.683 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.5 2023-12-19T18:45:01.919 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:02.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:01 smithi089 ceph-mon[88459]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:02.423 INFO:teuthology.orchestra.run.smithi059.stdout:115964117009 2023-12-19T18:45:02.424 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.3 2023-12-19T18:45:03.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:02 smithi059 ceph-mon[80661]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:03.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:02 smithi089 ceph-mon[88459]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:03.350 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.538+0000 7fa121c9e700 1 -- 172.21.15.59:0/2445038545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c0fbb40 msgr2=0x7fa11c0fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.538+0000 7fa121c9e700 1 --2- 172.21.15.59:0/2445038545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c0fbb40 0x7fa11c0fbfb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa110004940 tx=0x7fa110009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.538+0000 7fa121c9e700 1 -- 172.21.15.59:0/2445038545 shutdown_connections 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.538+0000 7fa121c9e700 1 --2- 172.21.15.59:0/2445038545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c0fbb40 0x7fa11c0fbfb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.538+0000 7fa121c9e700 1 --2- 172.21.15.59:0/2445038545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa11c101b00 0x7fa11c101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.538+0000 7fa121c9e700 1 -- 172.21.15.59:0/2445038545 >> 172.21.15.59:0/2445038545 conn(0x7fa11c0719b0 msgr2=0x7fa11c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.539+0000 7fa121c9e700 1 -- 172.21.15.59:0/2445038545 shutdown_connections 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.539+0000 7fa121c9e700 1 -- 172.21.15.59:0/2445038545 wait complete. 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.539+0000 7fa121c9e700 1 Processor -- start 2023-12-19T18:45:05.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.539+0000 7fa121c9e700 1 -- start start 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa121c9e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa11c0fbb40 0x7fa11c0fd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa121c9e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c101b00 0x7fa11c0fdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa121c9e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0fe260 con 0x7fa11c101b00 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa121c9e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0fe3d0 con 0x7fa11c0fbb40 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa11affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c101b00 0x7fa11c0fdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa11b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa11c0fbb40 0x7fa11c0fd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa11b7fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa11c0fbb40 0x7fa11c0fd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.59:38830/0 (socket says 172.21.15.59:38830) 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa11b7fe700 1 -- 172.21.15.59:0/86690301 learned_addr learned my addr 172.21.15.59:0/86690301 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa11affd700 1 -- 172.21.15.59:0/86690301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa11c0fbb40 msgr2=0x7fa11c0fd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:05.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa11affd700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa11c0fbb40 0x7fa11c0fd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa11affd700 1 -- 172.21.15.59:0/86690301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa110005040 con 0x7fa11c101b00 2023-12-19T18:45:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa11affd700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c101b00 0x7fa11c0fdc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa11000b0d0 tx=0x7fa110009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa103fff700 1 -- 172.21.15.59:0/86690301 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa110021070 con 0x7fa11c101b00 2023-12-19T18:45:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa103fff700 1 -- 172.21.15.59:0/86690301 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1100093b0 con 0x7fa11c101b00 2023-12-19T18:45:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c119800 con 0x7fa11c101b00 2023-12-19T18:45:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.540+0000 7fa103fff700 1 -- 172.21.15.59:0/86690301 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa11000e070 con 0x7fa11c101b00 2023-12-19T18:45:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.541+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c119c00 con 0x7fa11c101b00 2023-12-19T18:45:05.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.541+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa11c065da0 con 0x7fa11c101b00 2023-12-19T18:45:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.542+0000 7fa103fff700 1 -- 172.21.15.59:0/86690301 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa11001c600 con 0x7fa11c101b00 2023-12-19T18:45:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.542+0000 7fa103fff700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa104068f60 0x7fa10406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.543+0000 7fa103fff700 1 -- 172.21.15.59:0/86690301 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa1100884f0 con 0x7fa11c101b00 2023-12-19T18:45:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.543+0000 7fa11b7fe700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa104068f60 0x7fa10406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:05.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.543+0000 7fa103fff700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] conn(0x7fa10406ec30 0x7fa104071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:05.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.543+0000 7fa103fff700 1 -- 172.21.15.59:0/86690301 --> [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1040716f0 con 0x7fa10406ec30 2023-12-19T18:45:05.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.543+0000 7fa103fff700 1 -- 172.21.15.59:0/86690301 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa110088830 con 0x7fa11c101b00 2023-12-19T18:45:05.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.543+0000 7fa11bfff700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] conn(0x7fa10406ec30 0x7fa104071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:05.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.543+0000 7fa11b7fe700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa104068f60 0x7fa10406b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa10c008fd0 tx=0x7fa10c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:05.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.543+0000 7fa11bfff700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] conn(0x7fa10406ec30 0x7fa104071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:05.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.544+0000 7fa103fff700 1 -- 172.21.15.59:0/86690301 <== osd.2 v2:172.21.15.59:6818/240742436 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa1040716f0 con 0x7fa10406ec30 2023-12-19T18:45:05.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.559+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 --> [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa11c0fe590 con 0x7fa10406ec30 2023-12-19T18:45:05.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.559+0000 7fa103fff700 1 -- 172.21.15.59:0/86690301 <== osd.2 v2:172.21.15.59:6818/240742436 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa11c0fe590 con 0x7fa10406ec30 2023-12-19T18:45:05.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.559+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] conn(0x7fa10406ec30 msgr2=0x7fa104071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:05.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] conn(0x7fa10406ec30 0x7fa104071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa104068f60 msgr2=0x7fa10406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:05.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa104068f60 0x7fa10406b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa10c008fd0 tx=0x7fa10c00b040 comp rx=0 tx=0).stop 2023-12-19T18:45:05.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c101b00 msgr2=0x7fa11c0fdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:05.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c101b00 0x7fa11c0fdc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa11000b0d0 tx=0x7fa110009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:05.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 shutdown_connections 2023-12-19T18:45:05.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6818/240742436,v1:172.21.15.59:6819/240742436] conn(0x7fa10406ec30 0x7fa104071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa11c0fbb40 0x7fa11c0fd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa104068f60 0x7fa10406b410 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 --2- 172.21.15.59:0/86690301 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa11c101b00 0x7fa11c0fdc60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 >> 172.21.15.59:0/86690301 conn(0x7fa11c0719b0 msgr2=0x7fa11c065390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:05.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 shutdown_connections 2023-12-19T18:45:05.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.560+0000 7fa121c9e700 1 -- 172.21.15.59:0/86690301 wait complete. 2023-12-19T18:45:05.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.777+0000 7ffb33102700 1 -- 172.21.15.59:0/2929972453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 msgr2=0x7ffb2c103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:05.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.777+0000 7ffb33102700 1 --2- 172.21.15.59:0/2929972453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 0x7ffb2c103040 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009ce0 tx=0x7ffb24004d10 comp rx=0 tx=0).stop 2023-12-19T18:45:05.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.777+0000 7ffb33102700 1 -- 172.21.15.59:0/2929972453 shutdown_connections 2023-12-19T18:45:05.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.777+0000 7ffb33102700 1 --2- 172.21.15.59:0/2929972453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb2c0fb1e0 0x7ffb2c0fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.777+0000 7ffb33102700 1 --2- 172.21.15.59:0/2929972453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 0x7ffb2c103040 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.777+0000 7ffb33102700 1 -- 172.21.15.59:0/2929972453 >> 172.21.15.59:0/2929972453 conn(0x7ffb2c0711a0 msgr2=0x7ffb2c0715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:05.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.777+0000 7ffb33102700 1 -- 172.21.15.59:0/2929972453 shutdown_connections 2023-12-19T18:45:05.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.777+0000 7ffb33102700 1 -- 172.21.15.59:0/2929972453 wait complete. 2023-12-19T18:45:05.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb33102700 1 Processor -- start 2023-12-19T18:45:05.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb33102700 1 -- start start 2023-12-19T18:45:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb33102700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb2c0fb1e0 0x7ffb2c0fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb33102700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 0x7ffb2c0fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb33102700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c0fcf00 con 0x7ffb2c102c70 2023-12-19T18:45:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb33102700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c0fd070 con 0x7ffb2c0fb1e0 2023-12-19T18:45:05.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb2bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 0x7ffb2c0fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb2bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 0x7ffb2c0fc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35842/0 (socket says 172.21.15.59:35842) 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb2bfff700 1 -- 172.21.15.59:0/2276286091 learned_addr learned my addr 172.21.15.59:0/2276286091 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.778+0000 7ffb30e9e700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb2c0fb1e0 0x7ffb2c0fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.779+0000 7ffb2bfff700 1 -- 172.21.15.59:0/2276286091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb2c0fb1e0 msgr2=0x7ffb2c0fc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.779+0000 7ffb2bfff700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb2c0fb1e0 0x7ffb2c0fc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.779+0000 7ffb2bfff700 1 -- 172.21.15.59:0/2276286091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24009940 con 0x7ffb2c102c70 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.779+0000 7ffb2bfff700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 0x7ffb2c0fc900 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c002a50 tx=0x7ffb1c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.779+0000 7ffb297fa700 1 -- 172.21.15.59:0/2276286091 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb1c007cc0 con 0x7ffb2c102c70 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.779+0000 7ffb297fa700 1 -- 172.21.15.59:0/2276286091 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb1c00f040 con 0x7ffb2c102c70 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.779+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2c1199a0 con 0x7ffb2c102c70 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.779+0000 7ffb297fa700 1 -- 172.21.15.59:0/2276286091 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb1c00d9c0 con 0x7ffb2c102c70 2023-12-19T18:45:05.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.779+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2c119e80 con 0x7ffb2c102c70 2023-12-19T18:45:05.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.780+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2c0fff10 con 0x7ffb2c102c70 2023-12-19T18:45:05.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.781+0000 7ffb297fa700 1 -- 172.21.15.59:0/2276286091 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ffb1c00db40 con 0x7ffb2c102c70 2023-12-19T18:45:05.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.781+0000 7ffb297fa700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffb18068f60 0x7ffb1806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:05.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.781+0000 7ffb297fa700 1 -- 172.21.15.59:0/2276286091 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ffb1c086780 con 0x7ffb2c102c70 2023-12-19T18:45:05.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.781+0000 7ffb30e9e700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffb18068f60 0x7ffb1806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:05.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.782+0000 7ffb30e9e700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffb18068f60 0x7ffb1806b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb24003400 tx=0x7ffb24004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:05.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.786+0000 7ffb297fa700 1 -- 172.21.15.59:0/2276286091 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ffb1c052fc0 con 0x7ffb2c102c70 2023-12-19T18:45:05.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:05 smithi059 ceph-mon[80661]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:05.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.984+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ffb2c0fd230 con 0x7ffb2c102c70 2023-12-19T18:45:05.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.985+0000 7ffb297fa700 1 -- 172.21.15.59:0/2276286091 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffb2c0fd230 con 0x7ffb2c102c70 2023-12-19T18:45:05.986 INFO:teuthology.orchestra.run.smithi059.stdout:38654705698 2023-12-19T18:45:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffb18068f60 msgr2=0x7ffb1806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffb18068f60 0x7ffb1806b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb24003400 tx=0x7ffb24004d10 comp rx=0 tx=0).stop 2023-12-19T18:45:05.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 msgr2=0x7ffb2c0fc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 0x7ffb2c0fc900 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c002a50 tx=0x7ffb1c0046b0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 shutdown_connections 2023-12-19T18:45:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffb2c0fb1e0 0x7ffb2c0fc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffb18068f60 0x7ffb1806b410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 --2- 172.21.15.59:0/2276286091 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffb2c102c70 0x7ffb2c0fc900 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:05.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 >> 172.21.15.59:0/2276286091 conn(0x7ffb2c0711a0 msgr2=0x7ffb2c0fa210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:05.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.988+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 shutdown_connections 2023-12-19T18:45:05.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:05.989+0000 7ffb33102700 1 -- 172.21.15.59:0/2276286091 wait complete. 2023-12-19T18:45:05.991 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:05.992 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:05.993 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:06.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:05 smithi089 ceph-mon[88459]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:06.874 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:06 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2276286091' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-19T18:45:06.900 INFO:teuthology.orchestra.run.smithi059.stdout:81604378648 2023-12-19T18:45:06.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.2 2023-12-19T18:45:07.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:06 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/2276286091' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-19T18:45:07.480 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705697 got 38654705698 for osd.0 2023-12-19T18:45:07.480 DEBUG:teuthology.parallel:result is None 2023-12-19T18:45:07.716 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:07 smithi059 ceph-mon[80661]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:07.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:07 smithi089 ceph-mon[88459]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:08.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 -- 172.21.15.59:0/2436639736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb40fbb40 msgr2=0x7f1eb40fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:08.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 --2- 172.21.15.59:0/2436639736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb40fbb40 0x7f1eb40fbfb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c004940 tx=0x7f1e9c009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:08.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 -- 172.21.15.59:0/2436639736 shutdown_connections 2023-12-19T18:45:08.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 --2- 172.21.15.59:0/2436639736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb40fbb40 0x7f1eb40fbfb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:08.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 --2- 172.21.15.59:0/2436639736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1eb4101b00 0x7f1eb4101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 -- 172.21.15.59:0/2436639736 >> 172.21.15.59:0/2436639736 conn(0x7f1eb40719b0 msgr2=0x7f1eb4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 -- 172.21.15.59:0/2436639736 shutdown_connections 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 -- 172.21.15.59:0/2436639736 wait complete. 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 Processor -- start 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.547+0000 7f1eba105700 1 -- start start 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eba105700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1eb40fbb40 0x7f1eb4112640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eba105700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb4101b00 0x7f1eb4112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eba105700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb41131d0 con 0x7f1eb4101b00 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eba105700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb410c4c0 con 0x7f1eb40fbb40 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eabfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb4101b00 0x7f1eb4112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eabfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb4101b00 0x7f1eb4112b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35866/0 (socket says 172.21.15.59:35866) 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eabfff700 1 -- 172.21.15.59:0/1034748168 learned_addr learned my addr 172.21.15.59:0/1034748168 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eb37fe700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1eb40fbb40 0x7f1eb4112640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:08.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eabfff700 1 -- 172.21.15.59:0/1034748168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1eb40fbb40 msgr2=0x7f1eb4112640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eabfff700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1eb40fbb40 0x7f1eb4112640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eabfff700 1 -- 172.21.15.59:0/1034748168 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c005040 con 0x7f1eb4101b00 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eabfff700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb4101b00 0x7f1eb4112b80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009fd0 tx=0x7f1e9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.548+0000 7f1eb0ff9700 1 -- 172.21.15.59:0/1034748168 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e9c00e040 con 0x7f1eb4101b00 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.549+0000 7f1eb0ff9700 1 -- 172.21.15.59:0/1034748168 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e9c00bd90 con 0x7f1eb4101b00 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.549+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb410c740 con 0x7f1eb4101b00 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.549+0000 7f1eb0ff9700 1 -- 172.21.15.59:0/1034748168 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e9c013660 con 0x7f1eb4101b00 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.549+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb410cc20 con 0x7f1eb4101b00 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.550+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb41122d0 con 0x7f1eb4101b00 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.550+0000 7f1eb0ff9700 1 -- 172.21.15.59:0/1034748168 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1e9c01e420 con 0x7f1eb4101b00 2023-12-19T18:45:08.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.551+0000 7f1eb0ff9700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1ea0068f70 0x7f1ea006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:08.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.551+0000 7f1eb0ff9700 1 -- 172.21.15.59:0/1034748168 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1e9c08a870 con 0x7f1eb4101b00 2023-12-19T18:45:08.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.551+0000 7f1eb37fe700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1ea0068f70 0x7f1ea006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:08.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.551+0000 7f1eb37fe700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1ea0068f70 0x7f1ea006b420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4004f40 tx=0x7f1ea400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:08.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.555+0000 7f1eb0ff9700 1 -- 172.21.15.59:0/1034748168 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1e9c05aa90 con 0x7f1eb4101b00 2023-12-19T18:45:08.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:45:08.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.758+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1eb410dbe0 con 0x7f1eb4101b00 2023-12-19T18:45:08.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.759+0000 7f1eb0ff9700 1 -- 172.21.15.59:0/1034748168 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1e9c004db0 con 0x7f1eb4101b00 2023-12-19T18:45:08.760 INFO:teuthology.orchestra.run.smithi059.stdout:150323855375 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1ea0068f70 msgr2=0x7f1ea006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1ea0068f70 0x7f1ea006b420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4004f40 tx=0x7f1ea400b040 comp rx=0 tx=0).stop 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb4101b00 msgr2=0x7f1eb4112b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb4101b00 0x7f1eb4112b80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009fd0 tx=0x7f1e9c009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 shutdown_connections 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1eb40fbb40 0x7f1eb4112640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1ea0068f70 0x7f1ea006b420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 --2- 172.21.15.59:0/1034748168 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1eb4101b00 0x7f1eb4112b80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 >> 172.21.15.59:0/1034748168 conn(0x7f1eb40719b0 msgr2=0x7f1eb40fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 shutdown_connections 2023-12-19T18:45:08.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:08.761+0000 7f1eba105700 1 -- 172.21.15.59:0/1034748168 wait complete. 2023-12-19T18:45:09.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:08 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:45:09.935 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:09 smithi059 ceph-mon[80661]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:09.935 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:09 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1034748168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-19T18:45:09.965 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855372 got 150323855375 for osd.4 2023-12-19T18:45:09.965 DEBUG:teuthology.parallel:result is None 2023-12-19T18:45:10.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:09 smithi089 ceph-mon[88459]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:10.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:09 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1034748168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-19T18:45:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.761+0000 7fc192d9d700 1 -- 172.21.15.59:0/4241911614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c0fbb40 msgr2=0x7fc18c0fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.761+0000 7fc192d9d700 1 --2- 172.21.15.59:0/4241911614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c0fbb40 0x7fc18c0fbfb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc178004940 tx=0x7fc178009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.761+0000 7fc192d9d700 1 -- 172.21.15.59:0/4241911614 shutdown_connections 2023-12-19T18:45:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.761+0000 7fc192d9d700 1 --2- 172.21.15.59:0/4241911614 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c0fbb40 0x7fc18c0fbfb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:10.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.761+0000 7fc192d9d700 1 --2- 172.21.15.59:0/4241911614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc18c101b00 0x7fc18c101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:10.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.761+0000 7fc192d9d700 1 -- 172.21.15.59:0/4241911614 >> 172.21.15.59:0/4241911614 conn(0x7fc18c0719b0 msgr2=0x7fc18c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:10.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.761+0000 7fc192d9d700 1 -- 172.21.15.59:0/4241911614 shutdown_connections 2023-12-19T18:45:10.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.761+0000 7fc192d9d700 1 -- 172.21.15.59:0/4241911614 wait complete. 2023-12-19T18:45:10.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc192d9d700 1 Processor -- start 2023-12-19T18:45:10.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc192d9d700 1 -- start start 2023-12-19T18:45:10.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc192d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc18c0fbb40 0x7fc18c1135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:10.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc192d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c101b00 0x7fc18c113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:10.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc192d9d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18c114150 con 0x7fc18c101b00 2023-12-19T18:45:10.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc192d9d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18c10d480 con 0x7fc18c0fbb40 2023-12-19T18:45:10.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc18bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c101b00 0x7fc18c113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:10.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc18bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c101b00 0x7fc18c113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35884/0 (socket says 172.21.15.59:35884) 2023-12-19T18:45:10.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc190b39700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc18c0fbb40 0x7fc18c1135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:10.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.762+0000 7fc18bfff700 1 -- 172.21.15.59:0/3237685672 learned_addr learned my addr 172.21.15.59:0/3237685672 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:10.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.763+0000 7fc18bfff700 1 -- 172.21.15.59:0/3237685672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc18c0fbb40 msgr2=0x7fc18c1135c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:10.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.763+0000 7fc18bfff700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc18c0fbb40 0x7fc18c1135c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:10.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.763+0000 7fc18bfff700 1 -- 172.21.15.59:0/3237685672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc178005040 con 0x7fc18c101b00 2023-12-19T18:45:10.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.763+0000 7fc18bfff700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c101b00 0x7fc18c113b00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc178000c00 tx=0x7fc178009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:10.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.763+0000 7fc1897fa700 1 -- 172.21.15.59:0/3237685672 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc178021070 con 0x7fc18c101b00 2023-12-19T18:45:10.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.763+0000 7fc1897fa700 1 -- 172.21.15.59:0/3237685672 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc17801c400 con 0x7fc18c101b00 2023-12-19T18:45:10.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.763+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18c10d700 con 0x7fc18c101b00 2023-12-19T18:45:10.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.763+0000 7fc1897fa700 1 -- 172.21.15.59:0/3237685672 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc17800e070 con 0x7fc18c101b00 2023-12-19T18:45:10.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.763+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18c10dbe0 con 0x7fc18c101b00 2023-12-19T18:45:10.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.764+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18c0734d0 con 0x7fc18c101b00 2023-12-19T18:45:10.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.764+0000 7fc1897fa700 1 -- 172.21.15.59:0/3237685672 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc17801c970 con 0x7fc18c101b00 2023-12-19T18:45:10.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.765+0000 7fc1897fa700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fc17c068f60 0x7fc17c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:10.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.765+0000 7fc1897fa700 1 -- 172.21.15.59:0/3237685672 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc178087bb0 con 0x7fc18c101b00 2023-12-19T18:45:10.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.765+0000 7fc190b39700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fc17c068f60 0x7fc17c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:10.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.765+0000 7fc190b39700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fc17c068f60 0x7fc17c06b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc180009b80 tx=0x7fc18000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:10.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.769+0000 7fc1897fa700 1 -- 172.21.15.59:0/3237685672 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc17808e080 con 0x7fc18c101b00 2023-12-19T18:45:10.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.967+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc18c10eba0 con 0x7fc18c101b00 2023-12-19T18:45:10.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.967+0000 7fc1897fa700 1 -- 172.21.15.59:0/3237685672 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc178057e50 con 0x7fc18c101b00 2023-12-19T18:45:10.968 INFO:teuthology.orchestra.run.smithi059.stdout:115964117011 2023-12-19T18:45:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fc17c068f60 msgr2=0x7fc17c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fc17c068f60 0x7fc17c06b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc180009b80 tx=0x7fc18000b040 comp rx=0 tx=0).stop 2023-12-19T18:45:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c101b00 msgr2=0x7fc18c113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c101b00 0x7fc18c113b00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc178000c00 tx=0x7fc178009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 shutdown_connections 2023-12-19T18:45:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc18c0fbb40 0x7fc18c1135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:10.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fc17c068f60 0x7fc17c06b410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:10.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 --2- 172.21.15.59:0/3237685672 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc18c101b00 0x7fc18c113b00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:10.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 >> 172.21.15.59:0/3237685672 conn(0x7fc18c0719b0 msgr2=0x7fc18c065390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:10.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 shutdown_connections 2023-12-19T18:45:10.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:10.969+0000 7fc192d9d700 1 -- 172.21.15.59:0/3237685672 wait complete. 2023-12-19T18:45:11.743 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:11 smithi059 ceph-mon[80661]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:11.743 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:11 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3237685672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-19T18:45:11.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.950+0000 7fb41812f700 1 -- 172.21.15.59:0/3171939071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb410102c00 msgr2=0x7fb410102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:11.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.950+0000 7fb41812f700 1 --2- 172.21.15.59:0/3171939071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb410102c00 0x7fb410102fd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb408004940 tx=0x7fb408009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:11.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.950+0000 7fb41812f700 1 -- 172.21.15.59:0/3171939071 shutdown_connections 2023-12-19T18:45:11.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.950+0000 7fb41812f700 1 --2- 172.21.15.59:0/3171939071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4101035a0 0x7fb4100fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:11.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.950+0000 7fb41812f700 1 --2- 172.21.15.59:0/3171939071 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb410102c00 0x7fb410102fd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:11.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.950+0000 7fb41812f700 1 -- 172.21.15.59:0/3171939071 >> 172.21.15.59:0/3171939071 conn(0x7fb4100f8790 msgr2=0x7fb4100faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:11.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb41812f700 1 -- 172.21.15.59:0/3171939071 shutdown_connections 2023-12-19T18:45:11.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb41812f700 1 -- 172.21.15.59:0/3171939071 wait complete. 2023-12-19T18:45:11.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb41812f700 1 Processor -- start 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb41812f700 1 -- start start 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb41812f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb410102c00 0x7fb410113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb41812f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4101035a0 0x7fb410113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb41812f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb410114200 con 0x7fb4101035a0 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb41812f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41010d4f0 con 0x7fb410102c00 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb4156ca700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4101035a0 0x7fb410113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb4156ca700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4101035a0 0x7fb410113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44000/0 (socket says 172.21.15.59:44000) 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.951+0000 7fb4156ca700 1 -- 172.21.15.59:0/1462708673 learned_addr learned my addr 172.21.15.59:0/1462708673 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb415ecb700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb410102c00 0x7fb410113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb4156ca700 1 -- 172.21.15.59:0/1462708673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb410102c00 msgr2=0x7fb410113630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb4156ca700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb410102c00 0x7fb410113630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb4156ca700 1 -- 172.21.15.59:0/1462708673 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb408005040 con 0x7fb4101035a0 2023-12-19T18:45:11.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb4156ca700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4101035a0 0x7fb410113bb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb400002a50 tx=0x7fb400004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb4067fc700 1 -- 172.21.15.59:0/1462708673 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb400007cc0 con 0x7fb4101035a0 2023-12-19T18:45:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb4067fc700 1 -- 172.21.15.59:0/1462708673 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb40000f040 con 0x7fb4101035a0 2023-12-19T18:45:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb4067fc700 1 -- 172.21.15.59:0/1462708673 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb40000d920 con 0x7fb4101035a0 2023-12-19T18:45:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41010d780 con 0x7fb4101035a0 2023-12-19T18:45:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.952+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41010dcc0 con 0x7fb4101035a0 2023-12-19T18:45:11.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.953+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4100fff10 con 0x7fb4101035a0 2023-12-19T18:45:11.956 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.953+0000 7fb4067fc700 1 -- 172.21.15.59:0/1462708673 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb400009680 con 0x7fb4101035a0 2023-12-19T18:45:11.956 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.954+0000 7fb4067fc700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3fc068f60 0x7fb3fc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:11.956 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.954+0000 7fb4067fc700 1 -- 172.21.15.59:0/1462708673 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb4000866f0 con 0x7fb4101035a0 2023-12-19T18:45:11.956 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.954+0000 7fb415ecb700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3fc068f60 0x7fb3fc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:11.956 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.954+0000 7fb415ecb700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3fc068f60 0x7fb3fc06b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb40800b3b0 tx=0x7fb408009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:11.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:11.958+0000 7fb4067fc700 1 -- 172.21.15.59:0/1462708673 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb400056990 con 0x7fb4101035a0 2023-12-19T18:45:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:11 smithi089 ceph-mon[88459]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:11 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3237685672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-19T18:45:12.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.163+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb4100fc390 con 0x7fb4101035a0 2023-12-19T18:45:12.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.163+0000 7fb4067fc700 1 -- 172.21.15.59:0/1462708673 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb400056520 con 0x7fb4101035a0 2023-12-19T18:45:12.164 INFO:teuthology.orchestra.run.smithi059.stdout:171798691850 2023-12-19T18:45:12.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.165+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3fc068f60 msgr2=0x7fb3fc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:12.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.165+0000 7fb41812f700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3fc068f60 0x7fb3fc06b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb40800b3b0 tx=0x7fb408009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:12.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.165+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4101035a0 msgr2=0x7fb410113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:12.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.165+0000 7fb41812f700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4101035a0 0x7fb410113bb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb400002a50 tx=0x7fb400004600 comp rx=0 tx=0).stop 2023-12-19T18:45:12.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.165+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 shutdown_connections 2023-12-19T18:45:12.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.165+0000 7fb41812f700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb410102c00 0x7fb410113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:12.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.165+0000 7fb41812f700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3fc068f60 0x7fb3fc06b410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:12.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.165+0000 7fb41812f700 1 --2- 172.21.15.59:0/1462708673 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4101035a0 0x7fb410113bb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:12.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.166+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 >> 172.21.15.59:0/1462708673 conn(0x7fb4100f8790 msgr2=0x7fb4100fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:12.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.166+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 shutdown_connections 2023-12-19T18:45:12.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:12.166+0000 7fb41812f700 1 -- 172.21.15.59:0/1462708673 wait complete. 2023-12-19T18:45:12.973 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117009 got 115964117011 for osd.3 2023-12-19T18:45:12.974 DEBUG:teuthology.parallel:result is None 2023-12-19T18:45:13.137 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:13.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:12 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1462708673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-19T18:45:13.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:12 smithi059 ceph-mon[80661]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:13.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:12 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1462708673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-19T18:45:13.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:12 smithi089 ceph-mon[88459]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:14.718 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691848 got 171798691850 for osd.5 2023-12-19T18:45:14.718 DEBUG:teuthology.parallel:result is None 2023-12-19T18:45:15.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.181+0000 7f77a9526700 1 -- 172.21.15.59:0/977197597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 msgr2=0x7f77a4102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:15.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.181+0000 7f77a9526700 1 --2- 172.21.15.59:0/977197597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 0x7f77a4102fd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7798004940 tx=0x7f7798009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:15.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.181+0000 7f77a9526700 1 -- 172.21.15.59:0/977197597 shutdown_connections 2023-12-19T18:45:15.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.181+0000 7f77a9526700 1 --2- 172.21.15.59:0/977197597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77a41035a0 0x7f77a40fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:15.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.181+0000 7f77a9526700 1 --2- 172.21.15.59:0/977197597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 0x7f77a4102fd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:15.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.181+0000 7f77a9526700 1 -- 172.21.15.59:0/977197597 >> 172.21.15.59:0/977197597 conn(0x7f77a40f8790 msgr2=0x7f77a40faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:15.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.181+0000 7f77a9526700 1 -- 172.21.15.59:0/977197597 shutdown_connections 2023-12-19T18:45:15.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.181+0000 7f77a9526700 1 -- 172.21.15.59:0/977197597 wait complete. 2023-12-19T18:45:15.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a9526700 1 Processor -- start 2023-12-19T18:45:15.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a9526700 1 -- start start 2023-12-19T18:45:15.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a9526700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 0x7f77a4115830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:15.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a9526700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77a41035a0 0x7f77a4115d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:15.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a9526700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a41163c0 con 0x7f77a4102c00 2023-12-19T18:45:15.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a9526700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a410f6f0 con 0x7f77a41035a0 2023-12-19T18:45:15.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 0x7f77a4115830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 0x7f77a4115830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44024/0 (socket says 172.21.15.59:44024) 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a2ffd700 1 -- 172.21.15.59:0/201790002 learned_addr learned my addr 172.21.15.59:0/201790002 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a27fc700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77a41035a0 0x7f77a4115d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a2ffd700 1 -- 172.21.15.59:0/201790002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77a41035a0 msgr2=0x7f77a4115d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a2ffd700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77a41035a0 0x7f77a4115d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.182+0000 7f77a2ffd700 1 -- 172.21.15.59:0/201790002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798005040 con 0x7f77a4102c00 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.183+0000 7f77a2ffd700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 0x7f77a4115830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f779800b3b0 tx=0x7f7798009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.183+0000 7f778b7fe700 1 -- 172.21.15.59:0/201790002 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f779800e040 con 0x7f77a4102c00 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.183+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a410f970 con 0x7f77a4102c00 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.183+0000 7f778b7fe700 1 -- 172.21.15.59:0/201790002 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f779800bc50 con 0x7f77a4102c00 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.183+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a410feb0 con 0x7f77a4102c00 2023-12-19T18:45:15.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.183+0000 7f778b7fe700 1 -- 172.21.15.59:0/201790002 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7798013420 con 0x7f77a4102c00 2023-12-19T18:45:15.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.184+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a40fc390 con 0x7f77a4102c00 2023-12-19T18:45:15.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.184+0000 7f778b7fe700 1 -- 172.21.15.59:0/201790002 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f779801c430 con 0x7f77a4102c00 2023-12-19T18:45:15.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.185+0000 7f778b7fe700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f778c068f60 0x7f778c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:15.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.185+0000 7f778b7fe700 1 -- 172.21.15.59:0/201790002 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7798087850 con 0x7f77a4102c00 2023-12-19T18:45:15.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.185+0000 7f77a27fc700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f778c068f60 0x7f778c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:15.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.185+0000 7f77a27fc700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f778c068f60 0x7f778c06b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7794007fd0 tx=0x7f7794009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:15.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.189+0000 7f778b7fe700 1 -- 172.21.15.59:0/201790002 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7798057ba0 con 0x7f77a4102c00 2023-12-19T18:45:15.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.386+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f77a4110e10 con 0x7f77a4102c00 2023-12-19T18:45:15.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.387+0000 7f778b7fe700 1 -- 172.21.15.59:0/201790002 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7798057730 con 0x7f77a4102c00 2023-12-19T18:45:15.388 INFO:teuthology.orchestra.run.smithi059.stdout:60129542175 2023-12-19T18:45:15.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f778c068f60 msgr2=0x7f778c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:15.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f778c068f60 0x7f778c06b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7794007fd0 tx=0x7f7794009040 comp rx=0 tx=0).stop 2023-12-19T18:45:15.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 msgr2=0x7f77a4115830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:15.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 0x7f77a4115830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f779800b3b0 tx=0x7f7798009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:15.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 shutdown_connections 2023-12-19T18:45:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f778c068f60 0x7f778c06b410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a4102c00 0x7f77a4115830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 --2- 172.21.15.59:0/201790002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77a41035a0 0x7f77a4115d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 >> 172.21.15.59:0/201790002 conn(0x7f77a40f8790 msgr2=0x7f77a40fa060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 shutdown_connections 2023-12-19T18:45:15.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:15.389+0000 7f77a9526700 1 -- 172.21.15.59:0/201790002 wait complete. 2023-12-19T18:45:15.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:15 smithi059 ceph-mon[80661]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:15.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:15 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/201790002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-19T18:45:16.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:15 smithi089 ceph-mon[88459]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:16.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:15 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/201790002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-19T18:45:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.167+0000 7fb43e0b5700 1 -- 172.21.15.59:0/1235100886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb438102c70 msgr2=0x7fb438103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.167+0000 7fb43e0b5700 1 --2- 172.21.15.59:0/1235100886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb438102c70 0x7fb438103040 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb428004940 tx=0x7fb428009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.167+0000 7fb43e0b5700 1 -- 172.21.15.59:0/1235100886 shutdown_connections 2023-12-19T18:45:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.167+0000 7fb43e0b5700 1 --2- 172.21.15.59:0/1235100886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4380fb1e0 0x7fb4380fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.167+0000 7fb43e0b5700 1 --2- 172.21.15.59:0/1235100886 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb438102c70 0x7fb438103040 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.167+0000 7fb43e0b5700 1 -- 172.21.15.59:0/1235100886 >> 172.21.15.59:0/1235100886 conn(0x7fb4380711a0 msgr2=0x7fb4380715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.167+0000 7fb43e0b5700 1 -- 172.21.15.59:0/1235100886 shutdown_connections 2023-12-19T18:45:16.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.167+0000 7fb43e0b5700 1 -- 172.21.15.59:0/1235100886 wait complete. 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.168+0000 7fb43e0b5700 1 Processor -- start 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.168+0000 7fb43e0b5700 1 -- start start 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.168+0000 7fb43e0b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380fb1e0 0x7fb4381135f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.168+0000 7fb43e0b5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb438102c70 0x7fb438113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.168+0000 7fb43e0b5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43810d4b0 con 0x7fb4380fb1e0 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.168+0000 7fb43e0b5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43810d620 con 0x7fb438102c70 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb4377fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380fb1e0 0x7fb4381135f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb4377fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380fb1e0 0x7fb4381135f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44052/0 (socket says 172.21.15.59:44052) 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb4377fe700 1 -- 172.21.15.59:0/3431125819 learned_addr learned my addr 172.21.15.59:0/3431125819 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb436ffd700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb438102c70 0x7fb438113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb4377fe700 1 -- 172.21.15.59:0/3431125819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb438102c70 msgr2=0x7fb438113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb4377fe700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb438102c70 0x7fb438113b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:16.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb4377fe700 1 -- 172.21.15.59:0/3431125819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb428005040 con 0x7fb4380fb1e0 2023-12-19T18:45:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb4377fe700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380fb1e0 0x7fb4381135f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb428009180 tx=0x7fb428009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb41ffff700 1 -- 172.21.15.59:0/3431125819 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb42800e040 con 0x7fb4380fb1e0 2023-12-19T18:45:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43810d8a0 con 0x7fb4380fb1e0 2023-12-19T18:45:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb41ffff700 1 -- 172.21.15.59:0/3431125819 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb428004290 con 0x7fb4380fb1e0 2023-12-19T18:45:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb41ffff700 1 -- 172.21.15.59:0/3431125819 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb42801c3f0 con 0x7fb4380fb1e0 2023-12-19T18:45:16.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.169+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43810dde0 con 0x7fb4380fb1e0 2023-12-19T18:45:16.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.170+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb438072cc0 con 0x7fb4380fb1e0 2023-12-19T18:45:16.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.171+0000 7fb41ffff700 1 -- 172.21.15.59:0/3431125819 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb428013420 con 0x7fb4380fb1e0 2023-12-19T18:45:16.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.171+0000 7fb41ffff700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb420068f60 0x7fb42006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:16.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.171+0000 7fb41ffff700 1 -- 172.21.15.59:0/3431125819 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb428087e00 con 0x7fb4380fb1e0 2023-12-19T18:45:16.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.171+0000 7fb436ffd700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb420068f60 0x7fb42006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:16.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.172+0000 7fb436ffd700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb420068f60 0x7fb42006b410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb43810ee90 tx=0x7fb42c0091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:16.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.176+0000 7fb41ffff700 1 -- 172.21.15.59:0/3431125819 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb428058000 con 0x7fb4380fb1e0 2023-12-19T18:45:16.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.374+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb43804bbb0 con 0x7fb4380fb1e0 2023-12-19T18:45:16.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.374+0000 7fb41ffff700 1 -- 172.21.15.59:0/3431125819 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb428057b90 con 0x7fb4380fb1e0 2023-12-19T18:45:16.375 INFO:teuthology.orchestra.run.smithi059.stdout:81604378649 2023-12-19T18:45:16.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb420068f60 msgr2=0x7fb42006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb420068f60 0x7fb42006b410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb43810ee90 tx=0x7fb42c0091a0 comp rx=0 tx=0).stop 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380fb1e0 msgr2=0x7fb4381135f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380fb1e0 0x7fb4381135f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb428009180 tx=0x7fb428009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 shutdown_connections 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb420068f60 0x7fb42006b410 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380fb1e0 0x7fb4381135f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 --2- 172.21.15.59:0/3431125819 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb438102c70 0x7fb438113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 >> 172.21.15.59:0/3431125819 conn(0x7fb4380711a0 msgr2=0x7fb4380f9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 shutdown_connections 2023-12-19T18:45:16.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:16.377+0000 7fb43e0b5700 1 -- 172.21.15.59:0/3431125819 wait complete. 2023-12-19T18:45:16.770 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:16 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3431125819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-19T18:45:16.795 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542172 got 60129542175 for osd.1 2023-12-19T18:45:16.795 DEBUG:teuthology.parallel:result is None 2023-12-19T18:45:17.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:16 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3431125819' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-19T18:45:17.344 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378649 for osd.2 2023-12-19T18:45:17.344 DEBUG:teuthology.parallel:result is None 2023-12-19T18:45:17.344 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-12-19T18:45:17.345 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-19T18:45:17.621 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:17.850 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:17 smithi059 ceph-mon[80661]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:17.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:17 smithi089 ceph-mon[88459]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:19.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.104+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2006536398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 msgr2=0x7fb4f0101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.104+0000 7fb4f5ef5700 1 --2- 172.21.15.59:0/2006536398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 0x7fb4f0101ed0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0004940 tx=0x7fb4e0009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.104+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2006536398 shutdown_connections 2023-12-19T18:45:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.104+0000 7fb4f5ef5700 1 --2- 172.21.15.59:0/2006536398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f00fbb40 0x7fb4f00fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.104+0000 7fb4f5ef5700 1 --2- 172.21.15.59:0/2006536398 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 0x7fb4f0101ed0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.104+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2006536398 >> 172.21.15.59:0/2006536398 conn(0x7fb4f00719b0 msgr2=0x7fb4f0071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.104+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2006536398 shutdown_connections 2023-12-19T18:45:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.104+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2006536398 wait complete. 2023-12-19T18:45:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4f5ef5700 1 Processor -- start 2023-12-19T18:45:19.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4f5ef5700 1 -- start start 2023-12-19T18:45:19.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4f5ef5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f00fbb40 0x7fb4f010e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:19.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4f5ef5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 0x7fb4f010ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:19.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4f5ef5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f010f260 con 0x7fb4f0101b00 2023-12-19T18:45:19.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4f5ef5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f010f3a0 con 0x7fb4f00fbb40 2023-12-19T18:45:19.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4eeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 0x7fb4f010ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:19.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4eeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 0x7fb4f010ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44072/0 (socket says 172.21.15.59:44072) 2023-12-19T18:45:19.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4eeffd700 1 -- 172.21.15.59:0/2662955877 learned_addr learned my addr 172.21.15.59:0/2662955877 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:19.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.105+0000 7fb4ef7fe700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f00fbb40 0x7fb4f010e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:19.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.106+0000 7fb4eeffd700 1 -- 172.21.15.59:0/2662955877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f00fbb40 msgr2=0x7fb4f010e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:19.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.106+0000 7fb4eeffd700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f00fbb40 0x7fb4f010e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:19.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.106+0000 7fb4eeffd700 1 -- 172.21.15.59:0/2662955877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e0005040 con 0x7fb4f0101b00 2023-12-19T18:45:19.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.106+0000 7fb4eeffd700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 0x7fb4f010ec40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400b7c0 tx=0x7fb4e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:19.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.106+0000 7fb4d7fff700 1 -- 172.21.15.59:0/2662955877 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4e400f6f0 con 0x7fb4f0101b00 2023-12-19T18:45:19.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.106+0000 7fb4d7fff700 1 -- 172.21.15.59:0/2662955877 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4e400fd00 con 0x7fb4f0101b00 2023-12-19T18:45:19.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.106+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f01176d0 con 0x7fb4f0101b00 2023-12-19T18:45:19.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.106+0000 7fb4d7fff700 1 -- 172.21.15.59:0/2662955877 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4e4011e90 con 0x7fb4f0101b00 2023-12-19T18:45:19.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.106+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f0117c10 con 0x7fb4f0101b00 2023-12-19T18:45:19.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.107+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f0103fb0 con 0x7fb4f0101b00 2023-12-19T18:45:19.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.107+0000 7fb4d7fff700 1 -- 172.21.15.59:0/2662955877 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb4e400f850 con 0x7fb4f0101b00 2023-12-19T18:45:19.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.108+0000 7fb4d7fff700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb4d8068f60 0x7fb4d806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:19.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.108+0000 7fb4d7fff700 1 -- 172.21.15.59:0/2662955877 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb4e4086860 con 0x7fb4f0101b00 2023-12-19T18:45:19.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.108+0000 7fb4ef7fe700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb4d8068f60 0x7fb4d806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:19.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.108+0000 7fb4ef7fe700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb4d8068f60 0x7fb4d806b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0004f90 tx=0x7fb4e0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:19.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.112+0000 7fb4d7fff700 1 -- 172.21.15.59:0/2662955877 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb4e4053110 con 0x7fb4f0101b00 2023-12-19T18:45:19.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.305+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4f0065b70 con 0x7fb4d8068f60 2023-12-19T18:45:19.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.307+0000 7fb4d7fff700 1 -- 172.21.15.59:0/2662955877 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28137 (secure 0 0 0) 0x7fb4f0065b70 con 0x7fb4d8068f60 2023-12-19T18:45:19.308 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:45:19.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb4d8068f60 msgr2=0x7fb4d806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb4d8068f60 0x7fb4d806b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0004f90 tx=0x7fb4e0009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 msgr2=0x7fb4f010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 0x7fb4f010ec40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400b7c0 tx=0x7fb4e4009250 comp rx=0 tx=0).stop 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 shutdown_connections 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4f00fbb40 0x7fb4f010e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb4d8068f60 0x7fb4d806b410 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 --2- 172.21.15.59:0/2662955877 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4f0101b00 0x7fb4f010ec40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 >> 172.21.15.59:0/2662955877 conn(0x7fb4f00719b0 msgr2=0x7fb4f00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 shutdown_connections 2023-12-19T18:45:19.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:19.309+0000 7fb4f5ef5700 1 -- 172.21.15.59:0/2662955877 wait complete. 2023-12-19T18:45:19.311 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-12-19T18:45:19.929 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:19 smithi089 ceph-mon[88459]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:19.929 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:19 smithi059 ceph-mon[80661]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:19.960 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2023-12-19T18:45:18.534752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002503"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-12-19T18:44:30.035042+0000","last_change":"2023-12-19T18:44:05.925572+0000","last_active":"2023-12-19T18:44:30.035042+0000","last_peered":"2023-12-19T18:44:30.035042+0000","last_clean":"2023-12-19T18:44:30.035042+0000","last_became_active":"2023-12-19T18:44:05.925233+0000","last_became_peered":"2023-12-19T18:44:05.925233+0000","last_unstale":"2023-12-19T18:44:30.035042+0000","last_undegraded":"2023-12-19T18:44:30.035042+0000","last_fullsized":"2023-12-19T18:44:30.035042+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T18:43:21.313866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T18:43:21.313866+0000","last_clean_scrub_stamp":"2023-12-19T18:43:21.313866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:57:55.766741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3360000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.53100000000000003}]},{"osd":1,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.497},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.60899999999999999}]},{"osd":2,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.629}]},{"osd":3,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.68000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117013,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 18:44:45 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.35399999999999998}]},{"osd":1,"last update":"Tue Dec 19 18:44:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.439}]},{"osd":2,"last update":"Tue Dec 19 18:44:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40300000000000002}]},{"osd":4,"last update":"Tue Dec 19 18:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378650,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 18:44:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.308},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.36799999999999999}]},{"osd":1,"last update":"Tue Dec 19 18:44:21 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.39100000000000001}]},{"osd":3,"last update":"Tue Dec 19 18:44:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.73799999999999999}]},{"osd":4,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705701,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 18:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.34399999999999997,"15min":0.33900000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.373},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.52700000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.40600000000000003}]},{"osd":2,"last update":"Tue Dec 19 18:44:23 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Tue Dec 19 18:44:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.57799999999999996}]},{"osd":4,"last update":"Tue Dec 19 18:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.47599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542175,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.32800000000000001,"15min":0.32200000000000001},"min":{"1min":0.246,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.308},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.34899999999999998,"15min":0.34100000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.41799999999999998}]},{"osd":2,"last update":"Tue Dec 19 18:44:23 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.38400000000000001}]},{"osd":3,"last update":"Tue Dec 19 18:44:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.69399999999999995}]},{"osd":4,"last update":"Tue Dec 19 18:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T18:45:19.961 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-19T18:45:20.235 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:20.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:20 smithi059 ceph-mon[80661]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T18:45:20.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:20 smithi089 ceph-mon[88459]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T18:45:21.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 -- 172.21.15.59:0/922682445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29681035a0 msgr2=0x7f29680fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:21.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 --2- 172.21.15.59:0/922682445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29681035a0 0x7f29680fcc10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2950004940 tx=0x7f2950009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:21.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 -- 172.21.15.59:0/922682445 shutdown_connections 2023-12-19T18:45:21.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 --2- 172.21.15.59:0/922682445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29681035a0 0x7f29680fcc10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:21.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 --2- 172.21.15.59:0/922682445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2968102c00 0x7f2968102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:21.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 -- 172.21.15.59:0/922682445 >> 172.21.15.59:0/922682445 conn(0x7f29680f8790 msgr2=0x7f29680faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:21.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 -- 172.21.15.59:0/922682445 shutdown_connections 2023-12-19T18:45:21.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 -- 172.21.15.59:0/922682445 wait complete. 2023-12-19T18:45:21.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 Processor -- start 2023-12-19T18:45:21.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.826+0000 7f296d6c3700 1 -- start start 2023-12-19T18:45:21.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f296d6c3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2968102c00 0x7f2968113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:21.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f296d6c3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29681035a0 0x7f2968113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:21.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f296d6c3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29681141e0 con 0x7f2968102c00 2023-12-19T18:45:21.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f296d6c3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296810d4d0 con 0x7f29681035a0 2023-12-19T18:45:21.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f2966ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2968102c00 0x7f2968113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:21.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f2966ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2968102c00 0x7f2968113610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45320/0 (socket says 172.21.15.59:45320) 2023-12-19T18:45:21.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f2966ffd700 1 -- 172.21.15.59:0/3397045988 learned_addr learned my addr 172.21.15.59:0/3397045988 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:21.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f29667fc700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29681035a0 0x7f2968113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f2966ffd700 1 -- 172.21.15.59:0/3397045988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29681035a0 msgr2=0x7f2968113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f2966ffd700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29681035a0 0x7f2968113b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f2966ffd700 1 -- 172.21.15.59:0/3397045988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2950005040 con 0x7f2968102c00 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f2966ffd700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2968102c00 0x7f2968113610 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2958001d80 tx=0x7f2958009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f295f7fe700 1 -- 172.21.15.59:0/3397045988 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2958007d10 con 0x7f2968102c00 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.827+0000 7f295f7fe700 1 -- 172.21.15.59:0/3397045988 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2958007e70 con 0x7f2968102c00 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.828+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296810d760 con 0x7f2968102c00 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.828+0000 7f295f7fe700 1 -- 172.21.15.59:0/3397045988 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2958014640 con 0x7f2968102c00 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.828+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296810dca0 con 0x7f2968102c00 2023-12-19T18:45:21.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.828+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29681050d0 con 0x7f2968102c00 2023-12-19T18:45:21.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.829+0000 7f295f7fe700 1 -- 172.21.15.59:0/3397045988 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f295800e960 con 0x7f2968102c00 2023-12-19T18:45:21.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.829+0000 7f295f7fe700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2954068f60 0x7f295406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:21.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.829+0000 7f295f7fe700 1 -- 172.21.15.59:0/3397045988 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2958086790 con 0x7f2968102c00 2023-12-19T18:45:21.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.829+0000 7f29667fc700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2954068f60 0x7f295406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:21.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.830+0000 7f29667fc700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2954068f60 0x7f295406b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f295000b3b0 tx=0x7f2950009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:21.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:21.834+0000 7f295f7fe700 1 -- 172.21.15.59:0/3397045988 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f295804da00 con 0x7f2968102c00 2023-12-19T18:45:21.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:21 smithi059 ceph-mon[80661]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:22.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:21 smithi089 ceph-mon[88459]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:22.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.026+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f296810ebf0 con 0x7f2954068f60 2023-12-19T18:45:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.027+0000 7f295f7fe700 1 -- 172.21.15.59:0/3397045988 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28137 (secure 0 0 0) 0x7f296810ebf0 con 0x7f2954068f60 2023-12-19T18:45:22.028 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:45:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.029+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2954068f60 msgr2=0x7f295406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.029+0000 7f296d6c3700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2954068f60 0x7f295406b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f295000b3b0 tx=0x7f2950009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.029+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2968102c00 msgr2=0x7f2968113610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.029+0000 7f296d6c3700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2968102c00 0x7f2968113610 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2958001d80 tx=0x7f2958009ec0 comp rx=0 tx=0).stop 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.030+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 shutdown_connections 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.030+0000 7f296d6c3700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f2954068f60 0x7f295406b410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.030+0000 7f296d6c3700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2968102c00 0x7f2968113610 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.030+0000 7f296d6c3700 1 --2- 172.21.15.59:0/3397045988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29681035a0 0x7f2968113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.030+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 >> 172.21.15.59:0/3397045988 conn(0x7f29680f8790 msgr2=0x7f29680fa330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.030+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 shutdown_connections 2023-12-19T18:45:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:22.030+0000 7f296d6c3700 1 -- 172.21.15.59:0/3397045988 wait complete. 2023-12-19T18:45:22.032 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-12-19T18:45:22.742 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2023-12-19T18:45:20.535146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002535"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-12-19T18:44:30.035042+0000","last_change":"2023-12-19T18:44:05.925572+0000","last_active":"2023-12-19T18:44:30.035042+0000","last_peered":"2023-12-19T18:44:30.035042+0000","last_clean":"2023-12-19T18:44:30.035042+0000","last_became_active":"2023-12-19T18:44:05.925233+0000","last_became_peered":"2023-12-19T18:44:05.925233+0000","last_unstale":"2023-12-19T18:44:30.035042+0000","last_undegraded":"2023-12-19T18:44:30.035042+0000","last_fullsized":"2023-12-19T18:44:30.035042+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T18:43:21.313866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T18:43:21.313866+0000","last_clean_scrub_stamp":"2023-12-19T18:43:21.313866+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T04:57:55.766741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.3360000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.53100000000000003}]},{"osd":1,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.497},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.60899999999999999}]},{"osd":2,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.629}]},{"osd":3,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.68000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117013,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 18:44:45 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.35399999999999998}]},{"osd":1,"last update":"Tue Dec 19 18:44:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.439}]},{"osd":2,"last update":"Tue Dec 19 18:44:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.40300000000000002}]},{"osd":4,"last update":"Tue Dec 19 18:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.48499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 18:44:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.36199999999999999}]},{"osd":1,"last update":"Tue Dec 19 18:44:21 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.38600000000000001}]},{"osd":3,"last update":"Tue Dec 19 18:44:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53100000000000003}]},{"osd":4,"last update":"Tue Dec 19 18:45:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.51700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 18:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.34399999999999997,"15min":0.33900000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.52700000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.311}]},{"osd":2,"last update":"Tue Dec 19 18:44:23 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.35299999999999998}]},{"osd":3,"last update":"Tue Dec 19 18:44:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47199999999999998}]},{"osd":4,"last update":"Tue Dec 19 18:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.441},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542176,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 18:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.32800000000000001,"15min":0.32200000000000001},"min":{"1min":0.246,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.34899999999999998,"15min":0.34100000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.39500000000000002}]},{"osd":2,"last update":"Tue Dec 19 18:44:23 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.433},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.46999999999999997}]},{"osd":3,"last update":"Tue Dec 19 18:44:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55400000000000005}]},{"osd":4,"last update":"Tue Dec 19 18:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T18:45:22.743 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-12-19T18:45:22.743 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-19T18:45:22.743 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-12-19T18:45:22.744 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-19T18:45:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:23.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:22 smithi059 ceph-mon[80661]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T18:45:23.138 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:22 smithi059 ceph-mon[80661]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:23.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:22 smithi089 ceph-mon[88459]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T18:45:23.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:22 smithi089 ceph-mon[88459]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:24.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:45:24.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:45:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 -- 172.21.15.59:0/311548325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe730102c00 msgr2=0x7fe730102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 --2- 172.21.15.59:0/311548325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe730102c00 0x7fe730102fd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe720004940 tx=0x7fe720009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 -- 172.21.15.59:0/311548325 shutdown_connections 2023-12-19T18:45:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 --2- 172.21.15.59:0/311548325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7301035a0 0x7fe7300fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 --2- 172.21.15.59:0/311548325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe730102c00 0x7fe730102fd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 -- 172.21.15.59:0/311548325 >> 172.21.15.59:0/311548325 conn(0x7fe7300f8790 msgr2=0x7fe7300faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 -- 172.21.15.59:0/311548325 shutdown_connections 2023-12-19T18:45:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 -- 172.21.15.59:0/311548325 wait complete. 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 Processor -- start 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 -- start start 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.629+0000 7fe734db2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe730102c00 0x7fe73006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe734db2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7301035a0 0x7fe730070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe734db2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe730070a80 con 0x7fe7301035a0 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe734db2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe730070bf0 con 0x7fe730102c00 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe72dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7301035a0 0x7fe730070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe72dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7301035a0 0x7fe730070480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45342/0 (socket says 172.21.15.59:45342) 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe72dd9b700 1 -- 172.21.15.59:0/3678470518 learned_addr learned my addr 172.21.15.59:0/3678470518 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe72e59c700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe730102c00 0x7fe73006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:24.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe72dd9b700 1 -- 172.21.15.59:0/3678470518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe730102c00 msgr2=0x7fe73006ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:24.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe72dd9b700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe730102c00 0x7fe73006ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:24.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe72dd9b700 1 -- 172.21.15.59:0/3678470518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe720005040 con 0x7fe7301035a0 2023-12-19T18:45:24.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe72dd9b700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7301035a0 0x7fe730070480 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe724002a50 tx=0x7fe724004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:24.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe71effd700 1 -- 172.21.15.59:0/3678470518 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe724007cc0 con 0x7fe7301035a0 2023-12-19T18:45:24.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe730119870 con 0x7fe7301035a0 2023-12-19T18:45:24.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe71effd700 1 -- 172.21.15.59:0/3678470518 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe72400f040 con 0x7fe7301035a0 2023-12-19T18:45:24.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe71effd700 1 -- 172.21.15.59:0/3678470518 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe72400d920 con 0x7fe7301035a0 2023-12-19T18:45:24.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.630+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe730119d50 con 0x7fe7301035a0 2023-12-19T18:45:24.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.631+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7300fff10 con 0x7fe7301035a0 2023-12-19T18:45:24.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.632+0000 7fe71effd700 1 -- 172.21.15.59:0/3678470518 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe724009680 con 0x7fe7301035a0 2023-12-19T18:45:24.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.632+0000 7fe71effd700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe718068f60 0x7fe71806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:24.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.632+0000 7fe71effd700 1 -- 172.21.15.59:0/3678470518 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe7240867f0 con 0x7fe7301035a0 2023-12-19T18:45:24.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.632+0000 7fe72e59c700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe718068f60 0x7fe71806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:24.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.633+0000 7fe72e59c700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe718068f60 0x7fe71806b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe720004f90 tx=0x7fe720009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:24.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.637+0000 7fe71effd700 1 -- 172.21.15.59:0/3678470518 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe724052f80 con 0x7fe7301035a0 2023-12-19T18:45:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.876+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe7300fc390 con 0x7fe7301035a0 2023-12-19T18:45:24.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.877+0000 7fe71effd700 1 -- 172.21.15.59:0/3678470518 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe7240565a0 con 0x7fe7301035a0 2023-12-19T18:45:24.878 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:45:24.879 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-19T18:45:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe718068f60 msgr2=0x7fe71806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe718068f60 0x7fe71806b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe720004f90 tx=0x7fe720009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7301035a0 msgr2=0x7fe730070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7301035a0 0x7fe730070480 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe724002a50 tx=0x7fe724004600 comp rx=0 tx=0).stop 2023-12-19T18:45:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 shutdown_connections 2023-12-19T18:45:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe730102c00 0x7fe73006ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe718068f60 0x7fe71806b410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 --2- 172.21.15.59:0/3678470518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7301035a0 0x7fe730070480 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:24.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 >> 172.21.15.59:0/3678470518 conn(0x7fe7300f8790 msgr2=0x7fe7300fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 shutdown_connections 2023-12-19T18:45:24.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:24.880+0000 7fe734db2700 1 -- 172.21.15.59:0/3678470518 wait complete. 2023-12-19T18:45:25.201 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:24 smithi059 ceph-mon[80661]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:25.201 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:24 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3678470518' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-19T18:45:25.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:24 smithi089 ceph-mon[88459]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:25.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:24 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3678470518' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-19T18:45:25.644 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-12-19T18:45:25.644 INFO:tasks.cephadm:Setup complete, yielding 2023-12-19T18:45:25.645 INFO:teuthology.run_tasks:Running task print... 2023-12-19T18:45:25.654 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2023-12-19T18:45:25.654 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T18:45:25.820 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:45:25.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2023-12-19T18:45:26.100 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:27.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.716+0000 7f47691e2700 1 -- 172.21.15.59:0/3175774915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fde00 msgr2=0x7f47640fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:27.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.716+0000 7f47691e2700 1 --2- 172.21.15.59:0/3175774915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fde00 0x7f47640fe250 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4754004940 tx=0x7f4754009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:27.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.716+0000 7f47691e2700 1 -- 172.21.15.59:0/3175774915 shutdown_connections 2023-12-19T18:45:27.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.716+0000 7f47691e2700 1 --2- 172.21.15.59:0/3175774915 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fde00 0x7f47640fe250 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:27.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.716+0000 7f47691e2700 1 --2- 172.21.15.59:0/3175774915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47640fcbf0 0x7f47640fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:27.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.716+0000 7f47691e2700 1 -- 172.21.15.59:0/3175774915 >> 172.21.15.59:0/3175774915 conn(0x7f47640f82d0 msgr2=0x7f47640fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:27.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.716+0000 7f47691e2700 1 -- 172.21.15.59:0/3175774915 shutdown_connections 2023-12-19T18:45:27.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.716+0000 7f47691e2700 1 -- 172.21.15.59:0/3175774915 wait complete. 2023-12-19T18:45:27.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f47691e2700 1 Processor -- start 2023-12-19T18:45:27.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f47691e2700 1 -- start start 2023-12-19T18:45:27.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f47691e2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fcbf0 0x7f4764071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:27.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f47691e2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47640fde00 0x7f476406ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:27.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f47691e2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4764070590 con 0x7f47640fcbf0 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f47691e2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4764070700 con 0x7f47640fde00 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f4762d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fcbf0 0x7f4764071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f4762d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fcbf0 0x7f4764071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45362/0 (socket says 172.21.15.59:45362) 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f4762d9d700 1 -- 172.21.15.59:0/3821971466 learned_addr learned my addr 172.21.15.59:0/3821971466 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f476259c700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47640fde00 0x7f476406ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f4762d9d700 1 -- 172.21.15.59:0/3821971466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47640fde00 msgr2=0x7f476406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f4762d9d700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47640fde00 0x7f476406ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.717+0000 7f4762d9d700 1 -- 172.21.15.59:0/3821971466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4754005040 con 0x7f47640fcbf0 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.718+0000 7f4762d9d700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fcbf0 0x7f4764071b10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4758002a80 tx=0x7f4758004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.718+0000 7f474b7fe700 1 -- 172.21.15.59:0/3821971466 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4758007cc0 con 0x7f47640fcbf0 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.718+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47640709e0 con 0x7f47640fcbf0 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.718+0000 7f474b7fe700 1 -- 172.21.15.59:0/3821971466 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f475800f040 con 0x7f47640fcbf0 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.718+0000 7f474b7fe700 1 -- 172.21.15.59:0/3821971466 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f475800d9c0 con 0x7f47640fcbf0 2023-12-19T18:45:27.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.718+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47641155c0 con 0x7f47640fcbf0 2023-12-19T18:45:27.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.719+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476404bb80 con 0x7f47640fcbf0 2023-12-19T18:45:27.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.719+0000 7f474b7fe700 1 -- 172.21.15.59:0/3821971466 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f475800db20 con 0x7f47640fcbf0 2023-12-19T18:45:27.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.720+0000 7f474b7fe700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f474c068f60 0x7f474c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:27.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.720+0000 7f474b7fe700 1 -- 172.21.15.59:0/3821971466 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f47580867b0 con 0x7f47640fcbf0 2023-12-19T18:45:27.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.720+0000 7f476259c700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f474c068f60 0x7f474c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:27.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.720+0000 7f476259c700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f474c068f60 0x7f474c06b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f475400b3b0 tx=0x7f4754009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:27.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.724+0000 7f474b7fe700 1 -- 172.21.15.59:0/3821971466 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f47580569d0 con 0x7f47640fcbf0 2023-12-19T18:45:27.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:27 smithi089 ceph-mon[88459]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:27.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.918+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f4764105590 con 0x7f47640fcbf0 2023-12-19T18:45:27.919 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:27 smithi059 ceph-mon[80661]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:27.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.922+0000 7f474b7fe700 1 -- 172.21.15.59:0/3821971466 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f4758056560 con 0x7f47640fcbf0 2023-12-19T18:45:27.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.924+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f474c068f60 msgr2=0x7f474c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:27.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.924+0000 7f47691e2700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f474c068f60 0x7f474c06b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f475400b3b0 tx=0x7f4754009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:27.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.924+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fcbf0 msgr2=0x7f4764071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:27.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.924+0000 7f47691e2700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fcbf0 0x7f4764071b10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4758002a80 tx=0x7f4758004600 comp rx=0 tx=0).stop 2023-12-19T18:45:27.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.924+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 shutdown_connections 2023-12-19T18:45:27.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.924+0000 7f47691e2700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f474c068f60 0x7f474c06b410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:27.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.924+0000 7f47691e2700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47640fcbf0 0x7f4764071b10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:27.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.924+0000 7f47691e2700 1 --2- 172.21.15.59:0/3821971466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47640fde00 0x7f476406ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:27.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.924+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 >> 172.21.15.59:0/3821971466 conn(0x7f47640f82d0 msgr2=0x7f4764101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:27.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.925+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 shutdown_connections 2023-12-19T18:45:27.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:27.925+0000 7f47691e2700 1 -- 172.21.15.59:0/3821971466 wait complete. 2023-12-19T18:45:28.662 INFO:teuthology.run_tasks:Running task print... 2023-12-19T18:45:28.671 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2023-12-19T18:45:28.673 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T18:45:28.687 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:45:28.687 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph orch status' 2023-12-19T18:45:28.980 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:29.189 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:28 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3821971466' entity='client.admin' 2023-12-19T18:45:29.189 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:28 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:45:29.190 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:28 smithi059 ceph-mon[80661]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:29.190 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:28 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:29.190 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:28 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:45:29.190 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:28 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:29.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:28 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3821971466' entity='client.admin' 2023-12-19T18:45:29.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:28 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:45:29.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:28 smithi089 ceph-mon[88459]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:29.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:28 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:29.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:28 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:45:29.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:28 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.542+0000 7fb3bc42d700 1 -- 172.21.15.59:0/2494271626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 msgr2=0x7fb3b40fbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.542+0000 7fb3bc42d700 1 --2- 172.21.15.59:0/2494271626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 0x7fb3b40fbf10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0004940 tx=0x7fb3b0009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.542+0000 7fb3bc42d700 1 -- 172.21.15.59:0/2494271626 shutdown_connections 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.542+0000 7fb3bc42d700 1 --2- 172.21.15.59:0/2494271626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b40fcd10 0x7fb3b40fd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.542+0000 7fb3bc42d700 1 --2- 172.21.15.59:0/2494271626 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 0x7fb3b40fbf10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.542+0000 7fb3bc42d700 1 -- 172.21.15.59:0/2494271626 >> 172.21.15.59:0/2494271626 conn(0x7fb3b40f71c0 msgr2=0x7fb3b40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.542+0000 7fb3bc42d700 1 -- 172.21.15.59:0/2494271626 shutdown_connections 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.542+0000 7fb3bc42d700 1 -- 172.21.15.59:0/2494271626 wait complete. 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3bc42d700 1 Processor -- start 2023-12-19T18:45:30.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3bc42d700 1 -- start start 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3bc42d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 0x7fb3b40ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3bc42d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b40fcd10 0x7fb3b40fe050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3bc42d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b40fe6e0 con 0x7fb3b40fbb00 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3bc42d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b40fe850 con 0x7fb3b40fcd10 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3ba1c9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 0x7fb3b40ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3ba1c9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 0x7fb3b40ffbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45368/0 (socket says 172.21.15.59:45368) 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3ba1c9700 1 -- 172.21.15.59:0/3351828644 learned_addr learned my addr 172.21.15.59:0/3351828644 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3b99c8700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b40fcd10 0x7fb3b40fe050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3ba1c9700 1 -- 172.21.15.59:0/3351828644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b40fcd10 msgr2=0x7fb3b40fe050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3ba1c9700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b40fcd10 0x7fb3b40fe050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3ba1c9700 1 -- 172.21.15.59:0/3351828644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b0005040 con 0x7fb3b40fbb00 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.543+0000 7fb3ba1c9700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 0x7fb3b40ffbd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0004f90 tx=0x7fb3b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.544+0000 7fb3aaffd700 1 -- 172.21.15.59:0/3351828644 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3b000e040 con 0x7fb3b40fbb00 2023-12-19T18:45:30.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.544+0000 7fb3aaffd700 1 -- 172.21.15.59:0/3351828644 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3b000bcd0 con 0x7fb3b40fbb00 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.544+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b40fead0 con 0x7fb3b40fbb00 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.544+0000 7fb3aaffd700 1 -- 172.21.15.59:0/3351828644 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3b001c630 con 0x7fb3b40fbb00 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.544+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b41155a0 con 0x7fb3b40fbb00 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.545+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b404bb80 con 0x7fb3b40fbb00 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.545+0000 7fb3aaffd700 1 -- 172.21.15.59:0/3351828644 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb3b0013960 con 0x7fb3b40fbb00 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.546+0000 7fb3aaffd700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3a0068f60 0x7fb3a006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.546+0000 7fb3aaffd700 1 -- 172.21.15.59:0/3351828644 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb3b0087ef0 con 0x7fb3b40fbb00 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.546+0000 7fb3b99c8700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3a0068f60 0x7fb3a006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.546+0000 7fb3b99c8700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3a0068f60 0x7fb3a006b410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4006fd0 tx=0x7fb3a4008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:30.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.550+0000 7fb3aaffd700 1 -- 172.21.15.59:0/3351828644 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb3b0058190 con 0x7fb3b40fbb00 2023-12-19T18:45:30.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.763+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3b4066370 con 0x7fb3a0068f60 2023-12-19T18:45:30.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.764+0000 7fb3aaffd700 1 -- 172.21.15.59:0/3351828644 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb3b4066370 con 0x7fb3a0068f60 2023-12-19T18:45:30.764 INFO:teuthology.orchestra.run.smithi059.stdout:Backend: cephadm 2023-12-19T18:45:30.764 INFO:teuthology.orchestra.run.smithi059.stdout:Available: Yes 2023-12-19T18:45:30.764 INFO:teuthology.orchestra.run.smithi059.stdout:Paused: No 2023-12-19T18:45:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.765+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3a0068f60 msgr2=0x7fb3a006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.765+0000 7fb3bc42d700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3a0068f60 0x7fb3a006b410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4006fd0 tx=0x7fb3a4008040 comp rx=0 tx=0).stop 2023-12-19T18:45:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.765+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 msgr2=0x7fb3b40ffbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.765+0000 7fb3bc42d700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 0x7fb3b40ffbd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0004f90 tx=0x7fb3b0009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.765+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 shutdown_connections 2023-12-19T18:45:30.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.765+0000 7fb3bc42d700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb3a0068f60 0x7fb3a006b410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:30.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.765+0000 7fb3bc42d700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3b40fbb00 0x7fb3b40ffbd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:30.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.765+0000 7fb3bc42d700 1 --2- 172.21.15.59:0/3351828644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b40fcd10 0x7fb3b40fe050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:30.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.766+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 >> 172.21.15.59:0/3351828644 conn(0x7fb3b40f71c0 msgr2=0x7fb3b4065c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:30.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.766+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 shutdown_connections 2023-12-19T18:45:30.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:30.766+0000 7fb3bc42d700 1 -- 172.21.15.59:0/3351828644 wait complete. 2023-12-19T18:45:31.505 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph orch ps' 2023-12-19T18:45:31.780 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:31.901 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:31 smithi059 ceph-mon[80661]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:32.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:31 smithi089 ceph-mon[88459]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:32.896 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:32 smithi059 ceph-mon[80661]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:33.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:32 smithi089 ceph-mon[88459]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.236+0000 7fb075a3c700 1 -- 172.21.15.59:0/2220366295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 msgr2=0x7fb0700fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.236+0000 7fb075a3c700 1 --2- 172.21.15.59:0/2220366295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 0x7fb0700fd000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb060004940 tx=0x7fb060009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.236+0000 7fb075a3c700 1 -- 172.21.15.59:0/2220366295 shutdown_connections 2023-12-19T18:45:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.236+0000 7fb075a3c700 1 --2- 172.21.15.59:0/2220366295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0700fde00 0x7fb0700fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.236+0000 7fb075a3c700 1 --2- 172.21.15.59:0/2220366295 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 0x7fb0700fd000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.236+0000 7fb075a3c700 1 -- 172.21.15.59:0/2220366295 >> 172.21.15.59:0/2220366295 conn(0x7fb0700f82d0 msgr2=0x7fb0700fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.237+0000 7fb075a3c700 1 -- 172.21.15.59:0/2220366295 shutdown_connections 2023-12-19T18:45:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.237+0000 7fb075a3c700 1 -- 172.21.15.59:0/2220366295 wait complete. 2023-12-19T18:45:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.237+0000 7fb075a3c700 1 Processor -- start 2023-12-19T18:45:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.237+0000 7fb075a3c700 1 -- start start 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb075a3c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 0x7fb07010f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb075a3c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0700fde00 0x7fb07010fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb075a3c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0701102a0 con 0x7fb0700fcbf0 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb075a3c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb070110b40 con 0x7fb0700fde00 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb06effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 0x7fb07010f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb06effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 0x7fb07010f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49420/0 (socket says 172.21.15.59:49420) 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb06effd700 1 -- 172.21.15.59:0/2509149912 learned_addr learned my addr 172.21.15.59:0/2509149912 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb06e7fc700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0700fde00 0x7fb07010fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb06effd700 1 -- 172.21.15.59:0/2509149912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0700fde00 msgr2=0x7fb07010fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb06effd700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0700fde00 0x7fb07010fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb06effd700 1 -- 172.21.15.59:0/2509149912 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb060005040 con 0x7fb0700fcbf0 2023-12-19T18:45:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb06effd700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 0x7fb07010f760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb060004f90 tx=0x7fb060009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb0577fe700 1 -- 172.21.15.59:0/2509149912 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06000e040 con 0x7fb0700fcbf0 2023-12-19T18:45:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb070110dc0 con 0x7fb0700fcbf0 2023-12-19T18:45:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb0577fe700 1 -- 172.21.15.59:0/2509149912 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb06000bcd0 con 0x7fb0700fcbf0 2023-12-19T18:45:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb0577fe700 1 -- 172.21.15.59:0/2509149912 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06001c630 con 0x7fb0700fcbf0 2023-12-19T18:45:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.238+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb070111300 con 0x7fb0700fcbf0 2023-12-19T18:45:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.239+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb070101ea0 con 0x7fb0700fcbf0 2023-12-19T18:45:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.240+0000 7fb0577fe700 1 -- 172.21.15.59:0/2509149912 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb060013960 con 0x7fb0700fcbf0 2023-12-19T18:45:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.240+0000 7fb0577fe700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb058068f60 0x7fb05806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.240+0000 7fb0577fe700 1 -- 172.21.15.59:0/2509149912 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb060087ef0 con 0x7fb0700fcbf0 2023-12-19T18:45:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.240+0000 7fb06e7fc700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb058068f60 0x7fb05806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.241+0000 7fb06e7fc700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb058068f60 0x7fb05806b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb064004990 tx=0x7fb064009ee0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:33.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.245+0000 7fb0577fe700 1 -- 172.21.15.59:0/2509149912 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb060058190 con 0x7fb0700fcbf0 2023-12-19T18:45:33.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.448+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb070111570 con 0x7fb058068f60 2023-12-19T18:45:33.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.457+0000 7fb0577fe700 1 -- 172.21.15.59:0/2509149912 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7fb070111570 con 0x7fb058068f60 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (3m) 2m ago 5m 26.1M - 0.25.0 c8568f914cd2 528f56ce4fdd 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (5m) 2m ago 5m 6660k - 17.2.7-131-ga88f3edf f9ecdbaa1030 03ae34e0a398 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi089 smithi089 running (4m) 66s ago 4m 6665k - 17.2.7-131-ga88f3edf f9ecdbaa1030 8a39e517426d 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (3m) 2m ago 4m 82.3M - 9.4.7 954c08fa6188 a19de18455de 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.ccbnob smithi059 *:9283 running (6m) 2m ago 6m 469M - 17.2.7-131-ga88f3edf f9ecdbaa1030 9ce46f56adc1 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi089.isaaav smithi089 *:8443,9283 running (4m) 66s ago 4m 433M - 17.2.7-131-ga88f3edf f9ecdbaa1030 5890af67d417 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (6m) 2m ago 6m 44.0M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 297c7051711f 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi089 smithi089 running (4m) 66s ago 4m 38.4M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 5ac855c460fa 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (5m) 2m ago 5m 15.7M - 1.5.0 0da6a335fe13 bfe5bff75fbd 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi089 smithi089 *:9100 running (4m) 66s ago 4m 15.6M - 1.5.0 0da6a335fe13 057c3879e7a2 2023-12-19T18:45:33.458 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (3m) 2m ago 3m 37.9M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 87e75bce1c5b 2023-12-19T18:45:33.459 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (2m) 2m ago 2m 38.1M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 1b133ef58db0 2023-12-19T18:45:33.459 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (2m) 2m ago 2m 11.0M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 0f56936ea44c 2023-12-19T18:45:33.459 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi089 running (118s) 66s ago 118s 39.3M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 3eafaf27a901 2023-12-19T18:45:33.459 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi089 running (94s) 66s ago 94s 39.1M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 cc630810bb79 2023-12-19T18:45:33.459 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi089 running (70s) 66s ago 70s 10.7M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 f1d513a4069b 2023-12-19T18:45:33.459 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (3m) 2m ago 4m 35.8M - 2.43.0 a07b618ecd1d 5884a00b2f95 2023-12-19T18:45:33.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb058068f60 msgr2=0x7fb05806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:33.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb058068f60 0x7fb05806b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb064004990 tx=0x7fb064009ee0 comp rx=0 tx=0).stop 2023-12-19T18:45:33.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 msgr2=0x7fb07010f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:33.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 0x7fb07010f760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb060004f90 tx=0x7fb060009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:33.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 shutdown_connections 2023-12-19T18:45:33.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb058068f60 0x7fb05806b410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:33.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700fcbf0 0x7fb07010f760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:33.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 --2- 172.21.15.59:0/2509149912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0700fde00 0x7fb07010fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:33.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 >> 172.21.15.59:0/2509149912 conn(0x7fb0700f82d0 msgr2=0x7fb0700ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:33.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 shutdown_connections 2023-12-19T18:45:33.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:33.459+0000 7fb075a3c700 1 -- 172.21.15.59:0/2509149912 wait complete. 2023-12-19T18:45:33.904 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:33 smithi059 ceph-mon[80661]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:34.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:33 smithi089 ceph-mon[88459]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:34.049 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph orch ls' 2023-12-19T18:45:34.315 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:34.691 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:34 smithi059 ceph-mon[80661]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:35.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:34 smithi089 ceph-mon[88459]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:35.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:35 smithi059 ceph-mon[80661]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:35.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.917+0000 7f40fca44700 1 -- 172.21.15.59:0/3143154817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fff70 msgr2=0x7f40f81003e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:35.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.917+0000 7f40fca44700 1 --2- 172.21.15.59:0/3143154817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fff70 0x7f40f81003e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f40ec004940 tx=0x7f40ec009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:35.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 -- 172.21.15.59:0/3143154817 shutdown_connections 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 --2- 172.21.15.59:0/3143154817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fff70 0x7f40f81003e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 --2- 172.21.15.59:0/3143154817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f80fee00 0x7f40f80ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 -- 172.21.15.59:0/3143154817 >> 172.21.15.59:0/3143154817 conn(0x7f40f80fa5d0 msgr2=0x7f40f80fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 -- 172.21.15.59:0/3143154817 shutdown_connections 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 -- 172.21.15.59:0/3143154817 wait complete. 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 Processor -- start 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 -- start start 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fee00 0x7f40f810f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f80fff70 0x7f40f810fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8110250 con 0x7f40f80fee00 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40fca44700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8110af0 con 0x7f40f80fff70 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40f659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fee00 0x7f40f810f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.918+0000 7f40f659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fee00 0x7f40f810f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49442/0 (socket says 172.21.15.59:49442) 2023-12-19T18:45:35.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40f659c700 1 -- 172.21.15.59:0/2021511702 learned_addr learned my addr 172.21.15.59:0/2021511702 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:35.921 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40f5d9b700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f80fff70 0x7f40f810fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40f659c700 1 -- 172.21.15.59:0/2021511702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f80fff70 msgr2=0x7f40f810fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40f659c700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f80fff70 0x7f40f810fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40f659c700 1 -- 172.21.15.59:0/2021511702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec005040 con 0x7f40f80fee00 2023-12-19T18:45:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40f659c700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fee00 0x7f40f810f710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f40e800c830 tx=0x7f40e8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40e6ffd700 1 -- 172.21.15.59:0/2021511702 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40e80106f0 con 0x7f40f80fee00 2023-12-19T18:45:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40e6ffd700 1 -- 172.21.15.59:0/2021511702 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40e8010d00 con 0x7f40f80fee00 2023-12-19T18:45:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f8110dd0 con 0x7f40f80fee00 2023-12-19T18:45:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40e6ffd700 1 -- 172.21.15.59:0/2021511702 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40e8012e90 con 0x7f40f80fee00 2023-12-19T18:45:35.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.919+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f8111310 con 0x7f40f80fee00 2023-12-19T18:45:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.920+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f804bb80 con 0x7f40f80fee00 2023-12-19T18:45:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.921+0000 7f40e6ffd700 1 -- 172.21.15.59:0/2021511702 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f40e8010850 con 0x7f40f80fee00 2023-12-19T18:45:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.921+0000 7f40e6ffd700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f40e0068f60 0x7f40e006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.921+0000 7f40e6ffd700 1 -- 172.21.15.59:0/2021511702 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f40e80869d0 con 0x7f40f80fee00 2023-12-19T18:45:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.921+0000 7f40f5d9b700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f40e0068f60 0x7f40e006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.921+0000 7f40f5d9b700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f40e0068f60 0x7f40e006b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00b3b0 tx=0x7f40ec009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:35.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:35.925+0000 7f40e6ffd700 1 -- 172.21.15.59:0/2021511702 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f40e8056d10 con 0x7f40f80fee00 2023-12-19T18:45:36.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:35 smithi089 ceph-mon[88459]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.131+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f40f8111790 con 0x7f40e0068f60 2023-12-19T18:45:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.137+0000 7f40e6ffd700 1 -- 172.21.15.59:0/2021511702 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1161 (secure 0 0 0) 0x7f40f8111790 con 0x7f40e0068f60 2023-12-19T18:45:36.138 INFO:teuthology.orchestra.run.smithi059.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-12-19T18:45:36.138 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2023-12-19T18:45:36.139 INFO:teuthology.orchestra.run.smithi059.stdout:crash 2/2 2m ago 5m * 2023-12-19T18:45:36.139 INFO:teuthology.orchestra.run.smithi059.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2023-12-19T18:45:36.139 INFO:teuthology.orchestra.run.smithi059.stdout:mgr 2/2 2m ago 5m count:2 2023-12-19T18:45:36.139 INFO:teuthology.orchestra.run.smithi059.stdout:mon 2/2 2m ago 5m smithi059:172.21.15.59=smithi059;smithi089:172.21.15.89=smithi089;count:2 2023-12-19T18:45:36.139 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2023-12-19T18:45:36.139 INFO:teuthology.orchestra.run.smithi059.stdout:osd 6 2m ago - 2023-12-19T18:45:36.139 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2023-12-19T18:45:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f40e0068f60 msgr2=0x7f40e006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f40e0068f60 0x7f40e006b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00b3b0 tx=0x7f40ec009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fee00 msgr2=0x7f40f810f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fee00 0x7f40f810f710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f40e800c830 tx=0x7f40e8004470 comp rx=0 tx=0).stop 2023-12-19T18:45:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 shutdown_connections 2023-12-19T18:45:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f40e0068f60 0x7f40e006b410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40f80fee00 0x7f40f810f710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 --2- 172.21.15.59:0/2021511702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40f80fff70 0x7f40f810fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 >> 172.21.15.59:0/2021511702 conn(0x7f40f80fa5d0 msgr2=0x7f40f8103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.139+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 shutdown_connections 2023-12-19T18:45:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:36.140+0000 7f40fca44700 1 -- 172.21.15.59:0/2021511702 wait complete. 2023-12-19T18:45:36.771 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph orch host ls' 2023-12-19T18:45:37.040 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:37.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:37 smithi059 ceph-mon[80661]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:37.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:37 smithi059 ceph-mon[80661]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:37.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:37 smithi089 ceph-mon[88459]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:37.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:37 smithi089 ceph-mon[88459]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 -- 172.21.15.59:0/618002311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fcbf0 msgr2=0x7f871c0fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 --2- 172.21.15.59:0/618002311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fcbf0 0x7f871c0fd000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f870c004940 tx=0x7f870c009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 -- 172.21.15.59:0/618002311 shutdown_connections 2023-12-19T18:45:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 --2- 172.21.15.59:0/618002311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0fde00 0x7f871c0fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 --2- 172.21.15.59:0/618002311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fcbf0 0x7f871c0fd000 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:38.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 -- 172.21.15.59:0/618002311 >> 172.21.15.59:0/618002311 conn(0x7f871c0f82d0 msgr2=0x7f871c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 -- 172.21.15.59:0/618002311 shutdown_connections 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 -- 172.21.15.59:0/618002311 wait complete. 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 Processor -- start 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 -- start start 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0fcbf0 0x7f871c10f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fde00 0x7f871c10fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c110310 con 0x7f871c0fde00 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.454+0000 7f8720ef9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c110b60 con 0x7f871c0fcbf0 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f8719d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fde00 0x7f871c10fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f871a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0fcbf0 0x7f871c10f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f871a59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0fcbf0 0x7f871c10f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.59:48614/0 (socket says 172.21.15.59:48614) 2023-12-19T18:45:38.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f8719d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fde00 0x7f871c10fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49456/0 (socket says 172.21.15.59:49456) 2023-12-19T18:45:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f871a59c700 1 -- 172.21.15.59:0/3885269265 learned_addr learned my addr 172.21.15.59:0/3885269265 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f8719d9b700 1 -- 172.21.15.59:0/3885269265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0fcbf0 msgr2=0x7f871c10f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f8719d9b700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0fcbf0 0x7f871c10f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f8719d9b700 1 -- 172.21.15.59:0/3885269265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c005040 con 0x7f871c0fde00 2023-12-19T18:45:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f8719d9b700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fde00 0x7f871c10fd10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8710002820 tx=0x7f8710004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f870affd700 1 -- 172.21.15.59:0/3885269265 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8710007cc0 con 0x7f871c0fde00 2023-12-19T18:45:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f870affd700 1 -- 172.21.15.59:0/3885269265 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f871000f040 con 0x7f871c0fde00 2023-12-19T18:45:38.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c110e40 con 0x7f871c0fde00 2023-12-19T18:45:38.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f870affd700 1 -- 172.21.15.59:0/3885269265 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87100137d0 con 0x7f871c0fde00 2023-12-19T18:45:38.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.455+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c111380 con 0x7f871c0fde00 2023-12-19T18:45:38.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.456+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871c101ea0 con 0x7f871c0fde00 2023-12-19T18:45:38.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.457+0000 7f870affd700 1 -- 172.21.15.59:0/3885269265 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f871001c400 con 0x7f871c0fde00 2023-12-19T18:45:38.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.457+0000 7f870affd700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f8704068f70 0x7f870406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:38.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.457+0000 7f870affd700 1 -- 172.21.15.59:0/3885269265 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8710086650 con 0x7f871c0fde00 2023-12-19T18:45:38.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.457+0000 7f871a59c700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f8704068f70 0x7f870406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:38.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.458+0000 7f871a59c700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f8704068f70 0x7f870406b420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b3b0 tx=0x7f870c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:38.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.462+0000 7f870affd700 1 -- 172.21.15.59:0/3885269265 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f87100568f0 con 0x7f871c0fde00 2023-12-19T18:45:38.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.664+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f871c1115f0 con 0x7f8704068f70 2023-12-19T18:45:38.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.666+0000 7f870affd700 1 -- 172.21.15.59:0/3885269265 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+145 (secure 0 0 0) 0x7f871c1115f0 con 0x7f8704068f70 2023-12-19T18:45:38.666 INFO:teuthology.orchestra.run.smithi059.stdout:HOST ADDR LABELS STATUS 2023-12-19T18:45:38.667 INFO:teuthology.orchestra.run.smithi059.stdout:smithi059 172.21.15.59 2023-12-19T18:45:38.667 INFO:teuthology.orchestra.run.smithi059.stdout:smithi089 172.21.15.89 2023-12-19T18:45:38.667 INFO:teuthology.orchestra.run.smithi059.stdout:2 hosts in cluster 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f8704068f70 msgr2=0x7f870406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f8704068f70 0x7f870406b420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b3b0 tx=0x7f870c009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fde00 msgr2=0x7f871c10fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fde00 0x7f871c10fd10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8710002820 tx=0x7f8710004570 comp rx=0 tx=0).stop 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 shutdown_connections 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f871c0fcbf0 0x7f871c10f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f8704068f70 0x7f870406b420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 --2- 172.21.15.59:0/3885269265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f871c0fde00 0x7f871c10fd10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 >> 172.21.15.59:0/3885269265 conn(0x7f871c0f82d0 msgr2=0x7f871c0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:38.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 shutdown_connections 2023-12-19T18:45:38.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:38.668+0000 7f8720ef9700 1 -- 172.21.15.59:0/3885269265 wait complete. 2023-12-19T18:45:38.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:45:39.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:45:39.287 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph orch device ls' 2023-12-19T18:45:39.563 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:39.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:39 smithi089 ceph-mon[88459]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:39.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:39 smithi089 ceph-mon[88459]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:39.943 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:39 smithi059 ceph-mon[80661]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:39.944 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:39 smithi059 ceph-mon[80661]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:40.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.961+0000 7fe9e54c5700 1 -- 172.21.15.59:0/2513496246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fcbf0 msgr2=0x7fe9e00fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.961+0000 7fe9e54c5700 1 --2- 172.21.15.59:0/2513496246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fcbf0 0x7fe9e00fd000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4004940 tx=0x7fe9d4009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.961+0000 7fe9e54c5700 1 -- 172.21.15.59:0/2513496246 shutdown_connections 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.961+0000 7fe9e54c5700 1 --2- 172.21.15.59:0/2513496246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e00fde00 0x7fe9e00fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.961+0000 7fe9e54c5700 1 --2- 172.21.15.59:0/2513496246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fcbf0 0x7fe9e00fd000 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.961+0000 7fe9e54c5700 1 -- 172.21.15.59:0/2513496246 >> 172.21.15.59:0/2513496246 conn(0x7fe9e00f82d0 msgr2=0x7fe9e00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.961+0000 7fe9e54c5700 1 -- 172.21.15.59:0/2513496246 shutdown_connections 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.961+0000 7fe9e54c5700 1 -- 172.21.15.59:0/2513496246 wait complete. 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.962+0000 7fe9e54c5700 1 Processor -- start 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.962+0000 7fe9e54c5700 1 -- start start 2023-12-19T18:45:40.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.962+0000 7fe9e54c5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e00fcbf0 0x7fe9e0071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.962+0000 7fe9e54c5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fde00 0x7fe9e006ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.962+0000 7fe9e54c5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0070590 con 0x7fe9e00fde00 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.962+0000 7fe9e54c5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0070700 con 0x7fe9e00fcbf0 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.962+0000 7fe9de7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fde00 0x7fe9e006ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.962+0000 7fe9de7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fde00 0x7fe9e006ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49462/0 (socket says 172.21.15.59:49462) 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.962+0000 7fe9de7fc700 1 -- 172.21.15.59:0/3281399116 learned_addr learned my addr 172.21.15.59:0/3281399116 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9deffd700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e00fcbf0 0x7fe9e0071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9de7fc700 1 -- 172.21.15.59:0/3281399116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e00fcbf0 msgr2=0x7fe9e0071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9de7fc700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e00fcbf0 0x7fe9e0071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9de7fc700 1 -- 172.21.15.59:0/3281399116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d4005040 con 0x7fe9e00fde00 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9de7fc700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fde00 0x7fe9e006ff90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0002a50 tx=0x7fe9d00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9c77fe700 1 -- 172.21.15.59:0/3281399116 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9d0007cc0 con 0x7fe9e00fde00 2023-12-19T18:45:40.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9c77fe700 1 -- 172.21.15.59:0/3281399116 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9d000f040 con 0x7fe9e00fde00 2023-12-19T18:45:40.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9c77fe700 1 -- 172.21.15.59:0/3281399116 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9d000d9c0 con 0x7fe9e00fde00 2023-12-19T18:45:40.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e00709e0 con 0x7fe9e00fde00 2023-12-19T18:45:40.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.963+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e0115590 con 0x7fe9e00fde00 2023-12-19T18:45:40.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.964+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e004bb80 con 0x7fe9e00fde00 2023-12-19T18:45:40.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.965+0000 7fe9c77fe700 1 -- 172.21.15.59:0/3281399116 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe9d000db20 con 0x7fe9e00fde00 2023-12-19T18:45:40.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.965+0000 7fe9c77fe700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe9c8068f60 0x7fe9c806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:40.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.965+0000 7fe9c77fe700 1 -- 172.21.15.59:0/3281399116 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe9d00866a0 con 0x7fe9e00fde00 2023-12-19T18:45:40.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.965+0000 7fe9deffd700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe9c8068f60 0x7fe9c806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:40.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.966+0000 7fe9deffd700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe9c8068f60 0x7fe9c806b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400b7d0 tx=0x7fe9d4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:40.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:40.969+0000 7fe9c77fe700 1 -- 172.21.15.59:0/3281399116 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe9d0056940 con 0x7fe9e00fde00 2023-12-19T18:45:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.175+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9e0102750 con 0x7fe9c8068f60 2023-12-19T18:45:41.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.177+0000 7fe9c77fe700 1 -- 172.21.15.59:0/3281399116 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fe9e0102750 con 0x7fe9c8068f60 2023-12-19T18:45:41.178 INFO:teuthology.orchestra.run.smithi059.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-12-19T18:45:41.178 INFO:teuthology.orchestra.run.smithi059.stdout:smithi059 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009L400BGN 372G No 2m ago Has a FileSystem, LVM detected 2023-12-19T18:45:41.178 INFO:teuthology.orchestra.run.smithi059.stdout:smithi089 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040126400BGN 372G No 70s ago Has a FileSystem, LVM detected 2023-12-19T18:45:41.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe9c8068f60 msgr2=0x7fe9c806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:41.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe9c8068f60 0x7fe9c806b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400b7d0 tx=0x7fe9d4009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:41.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fde00 msgr2=0x7fe9e006ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:41.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fde00 0x7fe9e006ff90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0002a50 tx=0x7fe9d00046b0 comp rx=0 tx=0).stop 2023-12-19T18:45:41.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 shutdown_connections 2023-12-19T18:45:41.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9e00fcbf0 0x7fe9e0071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:41.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe9c8068f60 0x7fe9c806b410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:41.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 --2- 172.21.15.59:0/3281399116 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e00fde00 0x7fe9e006ff90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:41.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 >> 172.21.15.59:0/3281399116 conn(0x7fe9e00f82d0 msgr2=0x7fe9e0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:41.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 shutdown_connections 2023-12-19T18:45:41.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:41.179+0000 7fe9e54c5700 1 -- 172.21.15.59:0/3281399116 wait complete. 2023-12-19T18:45:41.895 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:41 smithi059 ceph-mon[80661]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:41.922 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T18:45:41.932 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:45:41.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2023-12-19T18:45:42.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:41 smithi089 ceph-mon[88459]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:42.197 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:42.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:42 smithi059 ceph-mon[80661]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:43.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:42 smithi089 ceph-mon[88459]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:43.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.787+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/2403609512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fbaf0 msgr2=0x7fbbbc0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:43.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.787+0000 7fbbc3ab0700 1 --2- 172.21.15.59:0/2403609512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fbaf0 0x7fbbbc0fbf00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009ce0 tx=0x7fbbb4004d10 comp rx=0 tx=0).stop 2023-12-19T18:45:43.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.787+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/2403609512 shutdown_connections 2023-12-19T18:45:43.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.787+0000 7fbbc3ab0700 1 --2- 172.21.15.59:0/2403609512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbbc0fcd00 0x7fbbbc0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:43.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.787+0000 7fbbc3ab0700 1 --2- 172.21.15.59:0/2403609512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fbaf0 0x7fbbbc0fbf00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:43.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.787+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/2403609512 >> 172.21.15.59:0/2403609512 conn(0x7fbbbc0f7190 msgr2=0x7fbbbc0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:43.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.787+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/2403609512 shutdown_connections 2023-12-19T18:45:43.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.787+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/2403609512 wait complete. 2023-12-19T18:45:43.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc3ab0700 1 Processor -- start 2023-12-19T18:45:43.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc3ab0700 1 -- start start 2023-12-19T18:45:43.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc3ab0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbbc0fbaf0 0x7fbbbc0ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:43.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc3ab0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fcd00 0x7fbbbc0fe040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:43.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc3ab0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc0fe640 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc3ab0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc0fe7b0 con 0x7fbbbc0fbaf0 2023-12-19T18:45:43.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc104b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fcd00 0x7fbbbc0fe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc104b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fcd00 0x7fbbbc0fe040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42542/0 (socket says 172.21.15.59:42542) 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc104b700 1 -- 172.21.15.59:0/3652792400 learned_addr learned my addr 172.21.15.59:0/3652792400 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc184c700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbbc0fbaf0 0x7fbbbc0ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc104b700 1 -- 172.21.15.59:0/3652792400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbbc0fbaf0 msgr2=0x7fbbbc0ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc104b700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbbc0fbaf0 0x7fbbbc0ffbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc104b700 1 -- 172.21.15.59:0/3652792400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4009940 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.788+0000 7fbbc104b700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fcd00 0x7fbbbc0fe040 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0002a50 tx=0x7fbbb00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.789+0000 7fbbba7fc700 1 -- 172.21.15.59:0/3652792400 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbb0007cc0 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.789+0000 7fbbba7fc700 1 -- 172.21.15.59:0/3652792400 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbb000f040 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.789+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc0fea90 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.789+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbbc115720 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.789+0000 7fbbba7fc700 1 -- 172.21.15.59:0/3652792400 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbb0009540 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.790+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbbc04bb80 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.790+0000 7fbbba7fc700 1 -- 172.21.15.59:0/3652792400 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbbb0009780 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.791+0000 7fbbba7fc700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbba4068f60 0x7fbba406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:43.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.791+0000 7fbbba7fc700 1 -- 172.21.15.59:0/3652792400 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbbb00868f0 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.791+0000 7fbbc184c700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbba4068f60 0x7fbba406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:43.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.791+0000 7fbbc184c700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbba4068f60 0x7fbba406b410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4015040 tx=0x7fbbb4004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:43.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:43.795+0000 7fbbba7fc700 1 -- 172.21.15.59:0/3652792400 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbbb0056a70 con 0x7fbbbc0fcd00 2023-12-19T18:45:43.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:43 smithi059 ceph-mon[80661]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:44.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:43 smithi089 ceph-mon[88459]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:44.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:44.018+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbbc065a80 con 0x7fbba4068f60 2023-12-19T18:45:44.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:44 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2023-12-19T18:45:45.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:44 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2023-12-19T18:45:45.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.651+0000 7fbbba7fc700 1 -- 172.21.15.59:0/3652792400 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbbbc065a80 con 0x7fbba4068f60 2023-12-19T18:45:45.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.654+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbba4068f60 msgr2=0x7fbba406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:45.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.654+0000 7fbbc3ab0700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbba4068f60 0x7fbba406b410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4015040 tx=0x7fbbb4004d10 comp rx=0 tx=0).stop 2023-12-19T18:45:45.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.654+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fcd00 msgr2=0x7fbbbc0fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:45.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.655+0000 7fbbc3ab0700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fcd00 0x7fbbbc0fe040 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0002a50 tx=0x7fbbb00046b0 comp rx=0 tx=0).stop 2023-12-19T18:45:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.655+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 shutdown_connections 2023-12-19T18:45:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.655+0000 7fbbc3ab0700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbbc0fbaf0 0x7fbbbc0ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.655+0000 7fbbc3ab0700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fbba4068f60 0x7fbba406b410 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:45.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.655+0000 7fbbc3ab0700 1 --2- 172.21.15.59:0/3652792400 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbbbc0fcd00 0x7fbbbc0fe040 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:45.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.655+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 >> 172.21.15.59:0/3652792400 conn(0x7fbbbc0f7190 msgr2=0x7fbbbc065370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:45.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.655+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 shutdown_connections 2023-12-19T18:45:45.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:45.655+0000 7fbbc3ab0700 1 -- 172.21.15.59:0/3652792400 wait complete. 2023-12-19T18:45:45.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:45 smithi059 ceph-mon[80661]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:45.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:45 smithi059 ceph-mon[80661]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:45.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2023-12-19T18:45:45.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:45 smithi059 ceph-mon[80661]: osdmap e42: 6 total, 6 up, 6 in 2023-12-19T18:45:45.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2023-12-19T18:45:45.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:45:45.630+0000 7f9271853700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T18:45:46.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:45 smithi089 ceph-mon[88459]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:45:46.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:45 smithi089 ceph-mon[88459]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:46.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2023-12-19T18:45:46.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:45 smithi089 ceph-mon[88459]: osdmap e42: 6 total, 6 up, 6 in 2023-12-19T18:45:46.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2023-12-19T18:45:46.316 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph fs dump' 2023-12-19T18:45:46.853 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2023-12-19T18:45:46.853 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: osdmap e43: 6 total, 6 up, 6 in 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: osdmap e44: 6 total, 6 up, 6 in 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: fsmap cephfs:0 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: Saving service mds.cephfs spec with placement count:4 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.abqklt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T18:45:46.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.abqklt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T18:45:46.855 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:46.855 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:46 smithi059 ceph-mon[80661]: Deploying daemon mds.cephfs.smithi059.abqklt on smithi059 2023-12-19T18:45:47.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2023-12-19T18:45:47.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: osdmap e43: 6 total, 6 up, 6 in 2023-12-19T18:45:47.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2023-12-19T18:45:47.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T18:45:47.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-19T18:45:47.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: osdmap e44: 6 total, 6 up, 6 in 2023-12-19T18:45:47.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2023-12-19T18:45:47.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: fsmap cephfs:0 2023-12-19T18:45:47.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: Saving service mds.cephfs spec with placement count:4 2023-12-19T18:45:47.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:47.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:45:47.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:47.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:45:47.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:47.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.abqklt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T18:45:47.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.abqklt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T18:45:47.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:47.007 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:46 smithi089 ceph-mon[88459]: Deploying daemon mds.cephfs.smithi059.abqklt on smithi059 2023-12-19T18:45:47.480 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:47.880 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:47 smithi059 ceph-mon[80661]: pgmap v171: 65 pgs: 16 creating+peering, 4 active+clean, 45 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:47.881 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:47 smithi059 ceph-mon[80661]: osdmap e45: 6 total, 6 up, 6 in 2023-12-19T18:45:48.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:47 smithi089 ceph-mon[88459]: pgmap v171: 65 pgs: 16 creating+peering, 4 active+clean, 45 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:48.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:47 smithi089 ceph-mon[88459]: osdmap e45: 6 total, 6 up, 6 in 2023-12-19T18:45:48.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:48 smithi059 ceph-mon[80661]: osdmap e46: 6 total, 6 up, 6 in 2023-12-19T18:45:48.929 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:48 smithi089 ceph-mon[88459]: osdmap e46: 6 total, 6 up, 6 in 2023-12-19T18:45:49.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.269+0000 7f0fa539e700 1 -- 172.21.15.59:0/3549796214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0066070 msgr2=0x7f0fa01019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:49.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.269+0000 7f0fa539e700 1 --2- 172.21.15.59:0/3549796214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0066070 0x7f0fa01019c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0f90004940 tx=0x7f0f90009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:49.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.269+0000 7f0fa539e700 1 -- 172.21.15.59:0/3549796214 shutdown_connections 2023-12-19T18:45:49.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.269+0000 7f0fa539e700 1 --2- 172.21.15.59:0/3549796214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa0101f00 0x7f0fa01042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.269+0000 7f0fa539e700 1 --2- 172.21.15.59:0/3549796214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0066070 0x7f0fa01019c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.269+0000 7f0fa539e700 1 -- 172.21.15.59:0/3549796214 >> 172.21.15.59:0/3549796214 conn(0x7f0fa00f6f70 msgr2=0x7f0fa00f93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.269+0000 7f0fa539e700 1 -- 172.21.15.59:0/3549796214 shutdown_connections 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.269+0000 7f0fa539e700 1 -- 172.21.15.59:0/3549796214 wait complete. 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.269+0000 7f0fa539e700 1 Processor -- start 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0fa539e700 1 -- start start 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0fa539e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa0066070 0x7f0fa010f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0fa539e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0101f00 0x7f0fa010fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0fa539e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa0110000 con 0x7f0fa0101f00 2023-12-19T18:45:49.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0fa539e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa01108a0 con 0x7f0fa0066070 2023-12-19T18:45:49.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0f9f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0101f00 0x7f0fa010fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:49.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0f9f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0101f00 0x7f0fa010fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42570/0 (socket says 172.21.15.59:42570) 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0f9f7fe700 1 -- 172.21.15.59:0/1062936715 learned_addr learned my addr 172.21.15.59:0/1062936715 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0f9ffff700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa0066070 0x7f0fa010f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0f9f7fe700 1 -- 172.21.15.59:0/1062936715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa0066070 msgr2=0x7f0fa010f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0f9f7fe700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa0066070 0x7f0fa010f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0f9f7fe700 1 -- 172.21.15.59:0/1062936715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f90005040 con 0x7f0fa0101f00 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.270+0000 7f0f9f7fe700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0101f00 0x7f0fa010fa00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0f88002a50 tx=0x7f0f880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.271+0000 7f0f9cff9700 1 -- 172.21.15.59:0/1062936715 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f88007cc0 con 0x7f0fa0101f00 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.271+0000 7f0f9cff9700 1 -- 172.21.15.59:0/1062936715 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f8800f040 con 0x7f0fa0101f00 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.271+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa0110b80 con 0x7f0fa0101f00 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.271+0000 7f0f9cff9700 1 -- 172.21.15.59:0/1062936715 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f8800da30 con 0x7f0fa0101f00 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.271+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa01110c0 con 0x7f0fa0101f00 2023-12-19T18:45:49.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.272+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa0109600 con 0x7f0fa0101f00 2023-12-19T18:45:49.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.272+0000 7f0f9cff9700 1 -- 172.21.15.59:0/1062936715 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0f8800db90 con 0x7f0fa0101f00 2023-12-19T18:45:49.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.273+0000 7f0f9cff9700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f8c068f60 0x7f0f8c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:49.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.273+0000 7f0f9cff9700 1 -- 172.21.15.59:0/1062936715 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0f88086930 con 0x7f0fa0101f00 2023-12-19T18:45:49.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.273+0000 7f0f9ffff700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f8c068f60 0x7f0f8c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:49.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.273+0000 7f0f9ffff700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f8c068f60 0x7f0f8c06b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0f90004f90 tx=0x7f0f90009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:49.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.278+0000 7f0f9cff9700 1 -- 172.21.15.59:0/1062936715 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0f88056890 con 0x7f0fa0101f00 2023-12-19T18:45:49.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.523+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0fa0111430 con 0x7f0fa0101f00 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.524+0000 7f0f9cff9700 1 -- 172.21.15.59:0/1062936715 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f0f88056420 con 0x7f0fa0101f00 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout:e2 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 2 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2023-12-19T18:45:49.525 INFO:teuthology.orchestra.run.smithi059.stdout:created 2023-12-19T18:45:45.631056+0000 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2023-12-19T18:45:45.631123+0000 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 0 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:in 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:up {} 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2023-12-19T18:45:49.526 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2023-12-19T18:45:49.527 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2023-12-19T18:45:49.527 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2023-12-19T18:45:49.527 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data disabled 2023-12-19T18:45:49.527 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2023-12-19T18:45:49.527 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 0 2023-12-19T18:45:49.527 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:45:49.527 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f8c068f60 msgr2=0x7f0f8c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f8c068f60 0x7f0f8c06b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0f90004f90 tx=0x7f0f90009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0101f00 msgr2=0x7f0fa010fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0101f00 0x7f0fa010fa00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0f88002a50 tx=0x7f0f880046b0 comp rx=0 tx=0).stop 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 shutdown_connections 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fa0066070 0x7f0fa010f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f0f8c068f60 0x7f0f8c06b410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 --2- 172.21.15.59:0/1062936715 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fa0101f00 0x7f0fa010fa00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 >> 172.21.15.59:0/1062936715 conn(0x7f0fa00f6f70 msgr2=0x7f0fa00f93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 shutdown_connections 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:49.526+0000 7f0fa539e700 1 -- 172.21.15.59:0/1062936715 wait complete. 2023-12-19T18:45:49.528 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 2 2023-12-19T18:45:49.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:49 smithi089 ceph-mon[88459]: pgmap v174: 65 pgs: 16 creating+peering, 19 active+clean, 30 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:49.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:49 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1062936715' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:45:49.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:49 smithi059 ceph-mon[80661]: pgmap v174: 65 pgs: 16 creating+peering, 19 active+clean, 30 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:45:49.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:49 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1062936715' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:45:50.401 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T18:45:50.410 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:45:50.410 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2023-12-19T18:45:50.683 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.59:6826/663831313,v1:172.21.15.59:6827/663831313] up:boot 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: daemon mds.cephfs.smithi059.abqklt assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: Cluster is now healthy 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: fsmap cephfs:0 1 up:standby 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.abqklt"}]: dispatch 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:creating} 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: daemon mds.cephfs.smithi059.abqklt is now active in filesystem cephfs as rank 0 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:50.726 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:50.727 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.dpgsbt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T18:45:50.727 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.dpgsbt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T18:45:50.727 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:50 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.59:6826/663831313,v1:172.21.15.59:6827/663831313] up:boot 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: daemon mds.cephfs.smithi059.abqklt assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: Cluster is now healthy 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: fsmap cephfs:0 1 up:standby 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.abqklt"}]: dispatch 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:creating} 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: daemon mds.cephfs.smithi059.abqklt is now active in filesystem cephfs as rank 0 2023-12-19T18:45:50.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:50.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:50.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:50.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.dpgsbt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T18:45:50.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.dpgsbt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T18:45:50.950 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:50 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:51.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:51 smithi089 ceph-mon[88459]: Deploying daemon mds.cephfs.smithi089.dpgsbt on smithi089 2023-12-19T18:45:51.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:51 smithi089 ceph-mon[88459]: pgmap v175: 65 pgs: 3 creating+peering, 62 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2023-12-19T18:45:51.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:51 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.59:6826/663831313,v1:172.21.15.59:6827/663831313] up:active 2023-12-19T18:45:51.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:51 smithi089 ceph-mon[88459]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 2023-12-19T18:45:51.756 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:51 smithi059 ceph-mon[80661]: Deploying daemon mds.cephfs.smithi089.dpgsbt on smithi089 2023-12-19T18:45:51.756 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:51 smithi059 ceph-mon[80661]: pgmap v175: 65 pgs: 3 creating+peering, 62 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2023-12-19T18:45:51.756 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:51 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.59:6826/663831313,v1:172.21.15.59:6827/663831313] up:active 2023-12-19T18:45:51.756 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:51 smithi059 ceph-mon[80661]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 2023-12-19T18:45:52.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.299+0000 7fcbdabb8700 1 -- 172.21.15.59:0/6676008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd40fdb80 msgr2=0x7fcbd4104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:52.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.299+0000 7fcbdabb8700 1 --2- 172.21.15.59:0/6676008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd40fdb80 0x7fcbd4104440 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0004940 tx=0x7fcbc0009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:52.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.299+0000 7fcbdabb8700 1 -- 172.21.15.59:0/6676008 shutdown_connections 2023-12-19T18:45:52.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.299+0000 7fcbdabb8700 1 --2- 172.21.15.59:0/6676008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd40fdb80 0x7fcbd4104440 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:52.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.299+0000 7fcbdabb8700 1 --2- 172.21.15.59:0/6676008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd40fd1a0 0x7fcbd40fd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:52.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.299+0000 7fcbdabb8700 1 -- 172.21.15.59:0/6676008 >> 172.21.15.59:0/6676008 conn(0x7fcbd40f6950 msgr2=0x7fcbd40f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:52.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.300+0000 7fcbdabb8700 1 -- 172.21.15.59:0/6676008 shutdown_connections 2023-12-19T18:45:52.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.300+0000 7fcbdabb8700 1 -- 172.21.15.59:0/6676008 wait complete. 2023-12-19T18:45:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.300+0000 7fcbdabb8700 1 Processor -- start 2023-12-19T18:45:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.300+0000 7fcbdabb8700 1 -- start start 2023-12-19T18:45:52.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbdabb8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd40fd1a0 0x7fcbd410f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbdabb8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd410fd60 0x7fcbd4110c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbdabb8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd4110220 con 0x7fcbd410fd60 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbdabb8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd4110360 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbd3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd410fd60 0x7fcbd4110c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbd8954700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd40fd1a0 0x7fcbd410f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbd3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd410fd60 0x7fcbd4110c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60234/0 (socket says 172.21.15.59:60234) 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbd3fff700 1 -- 172.21.15.59:0/806098201 learned_addr learned my addr 172.21.15.59:0/806098201 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbd8954700 1 -- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd410fd60 msgr2=0x7fcbd4110c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbd8954700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd410fd60 0x7fcbd4110c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbd8954700 1 -- 172.21.15.59:0/806098201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc0005040 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbd3fff700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd410fd60 0x7fcbd4110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.301+0000 7fcbd8954700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd40fd1a0 0x7fcbd410f820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800c760 tx=0x7fcbc80043c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.302+0000 7fcbd17fa700 1 -- 172.21.15.59:0/806098201 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbc80106f0 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.302+0000 7fcbd17fa700 1 -- 172.21.15.59:0/806098201 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbc8010d00 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.302+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd4111260 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.302+0000 7fcbd17fa700 1 -- 172.21.15.59:0/806098201 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbc8012dd0 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.302+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd4111770 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.303+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd4109830 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.303+0000 7fcbd17fa700 1 -- 172.21.15.59:0/806098201 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcbc801b440 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.304+0000 7fcbd17fa700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcbc4068f60 0x7fcbc406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.304+0000 7fcbd17fa700 1 -- 172.21.15.59:0/806098201 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fcbc8086cd0 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.304+0000 7fcbd3fff700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcbc4068f60 0x7fcbc406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:52.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.304+0000 7fcbd3fff700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcbc4068f60 0x7fcbc406b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0004f90 tx=0x7fcbc0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:52.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.308+0000 7fcbd17fa700 1 -- 172.21.15.59:0/806098201 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcbc8056b90 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.561+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fcbd4107970 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.944+0000 7fcbd17fa700 1 -- 172.21.15.59:0/806098201 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fcbc8056720 con 0x7fcbd40fd1a0 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.946+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcbc4068f60 msgr2=0x7fcbc406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.946+0000 7fcbdabb8700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcbc4068f60 0x7fcbc406b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0004f90 tx=0x7fcbc0009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.946+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd40fd1a0 msgr2=0x7fcbd410f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.946+0000 7fcbdabb8700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd40fd1a0 0x7fcbd410f820 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800c760 tx=0x7fcbc80043c0 comp rx=0 tx=0).stop 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.947+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 shutdown_connections 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.947+0000 7fcbdabb8700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbd40fd1a0 0x7fcbd410f820 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.947+0000 7fcbdabb8700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcbc4068f60 0x7fcbc406b410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.947+0000 7fcbdabb8700 1 --2- 172.21.15.59:0/806098201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcbd410fd60 0x7fcbd4110c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.947+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 >> 172.21.15.59:0/806098201 conn(0x7fcbd40f6950 msgr2=0x7fcbd40f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.947+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 shutdown_connections 2023-12-19T18:45:52.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:52.947+0000 7fcbdabb8700 1 -- 172.21.15.59:0/806098201 wait complete. 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.rtqzmc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.rtqzmc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: Deploying daemon mds.cephfs.smithi059.rtqzmc on smithi059 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: pgmap v176: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/806098201' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.89:6824/1622105230,v1:172.21.15.89:6825/1622105230] up:boot 2023-12-19T18:45:53.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2023-12-19T18:45:53.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 1 up:standby 2023-12-19T18:45:53.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.dpgsbt"}]: dispatch 2023-12-19T18:45:53.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:53 smithi059 ceph-mon[80661]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 1 up:standby 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.rtqzmc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi059.rtqzmc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: Deploying daemon mds.cephfs.smithi059.rtqzmc on smithi059 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: pgmap v176: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/806098201' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2023-12-19T18:45:53.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.89:6824/1622105230,v1:172.21.15.89:6825/1622105230] up:boot 2023-12-19T18:45:53.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2023-12-19T18:45:53.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 1 up:standby 2023-12-19T18:45:53.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.dpgsbt"}]: dispatch 2023-12-19T18:45:53.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:53 smithi089 ceph-mon[88459]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 1 up:standby 2023-12-19T18:45:53.679 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T18:45:53.689 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:45:53.690 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2023-12-19T18:45:54.312 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:54.383 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:54 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:45:54.424 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:54 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:45:55.435 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:55 smithi059 ceph-mon[80661]: pgmap v177: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2023-12-19T18:45:55.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:55 smithi089 ceph-mon[88459]: pgmap v177: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2023-12-19T18:45:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.894+0000 7fe2dcf57700 1 -- 172.21.15.59:0/225261544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d8066090 msgr2=0x7fe2d80ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.894+0000 7fe2dcf57700 1 --2- 172.21.15.59:0/225261544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d8066090 0x7fe2d80ff7a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8004940 tx=0x7fe2c8009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.894+0000 7fe2dcf57700 1 -- 172.21.15.59:0/225261544 shutdown_connections 2023-12-19T18:45:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.894+0000 7fe2dcf57700 1 --2- 172.21.15.59:0/225261544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d80ffd70 0x7fe2d81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.894+0000 7fe2dcf57700 1 --2- 172.21.15.59:0/225261544 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d8066090 0x7fe2d80ff7a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.894+0000 7fe2dcf57700 1 -- 172.21.15.59:0/225261544 >> 172.21.15.59:0/225261544 conn(0x7fe2d80f7190 msgr2=0x7fe2d80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:55.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.894+0000 7fe2dcf57700 1 -- 172.21.15.59:0/225261544 shutdown_connections 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.894+0000 7fe2dcf57700 1 -- 172.21.15.59:0/225261544 wait complete. 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2dcf57700 1 Processor -- start 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2dcf57700 1 -- start start 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2dcf57700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d8066090 0x7fe2d80fd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2dcf57700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d80ffd70 0x7fe2d80fb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2dcf57700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80fbca0 con 0x7fe2d80ffd70 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2dcf57700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d80fbe10 con 0x7fe2d8066090 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2d5d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d80ffd70 0x7fe2d80fb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2d5d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d80ffd70 0x7fe2d80fb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60252/0 (socket says 172.21.15.59:60252) 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2d5d9b700 1 -- 172.21.15.59:0/4195745283 learned_addr learned my addr 172.21.15.59:0/4195745283 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2d659c700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d8066090 0x7fe2d80fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2d5d9b700 1 -- 172.21.15.59:0/4195745283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d8066090 msgr2=0x7fe2d80fd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2d5d9b700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d8066090 0x7fe2d80fd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:55.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.895+0000 7fe2d5d9b700 1 -- 172.21.15.59:0/4195745283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c8005040 con 0x7fe2d80ffd70 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.896+0000 7fe2d5d9b700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d80ffd70 0x7fe2d80fb6a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc002a80 tx=0x7fe2cc004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.896+0000 7fe2c6ffd700 1 -- 172.21.15.59:0/4195745283 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2cc004e80 con 0x7fe2d80ffd70 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.896+0000 7fe2c6ffd700 1 -- 172.21.15.59:0/4195745283 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2cc00f040 con 0x7fe2d80ffd70 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.896+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d80fc0f0 con 0x7fe2d80ffd70 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.896+0000 7fe2c6ffd700 1 -- 172.21.15.59:0/4195745283 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2cc00da00 con 0x7fe2d80ffd70 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.896+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d81155e0 con 0x7fe2d80ffd70 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.897+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d8107610 con 0x7fe2d80ffd70 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.897+0000 7fe2c6ffd700 1 -- 172.21.15.59:0/4195745283 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe2cc007cc0 con 0x7fe2d80ffd70 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.898+0000 7fe2c6ffd700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe2c0068f60 0x7fe2c006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.898+0000 7fe2c6ffd700 1 -- 172.21.15.59:0/4195745283 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe2cc0863f0 con 0x7fe2d80ffd70 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.898+0000 7fe2d659c700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe2c0068f60 0x7fe2c006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.898+0000 7fe2d659c700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe2c0068f60 0x7fe2c006b410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8004890 tx=0x7fe2c8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:55.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:55.902+0000 7fe2c6ffd700 1 -- 172.21.15.59:0/4195745283 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe2cc04ca00 con 0x7fe2d80ffd70 2023-12-19T18:45:56.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.169+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe2d802b150 con 0x7fe2d80ffd70 2023-12-19T18:45:56.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.223+0000 7fe2c6ffd700 1 -- 172.21.15.59:0/4195745283 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe2cc0562d0 con 0x7fe2d80ffd70 2023-12-19T18:45:56.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.225+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe2c0068f60 msgr2=0x7fe2c006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:56.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.225+0000 7fe2dcf57700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe2c0068f60 0x7fe2c006b410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8004890 tx=0x7fe2c8009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:56.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.225+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d80ffd70 msgr2=0x7fe2d80fb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:56.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.225+0000 7fe2dcf57700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d80ffd70 0x7fe2d80fb6a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc002a80 tx=0x7fe2cc004630 comp rx=0 tx=0).stop 2023-12-19T18:45:56.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.225+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 shutdown_connections 2023-12-19T18:45:56.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.226+0000 7fe2dcf57700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe2d8066090 0x7fe2d80fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:56.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.226+0000 7fe2dcf57700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fe2c0068f60 0x7fe2c006b410 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:56.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.226+0000 7fe2dcf57700 1 --2- 172.21.15.59:0/4195745283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2d80ffd70 0x7fe2d80fb6a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:56.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.226+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 >> 172.21.15.59:0/4195745283 conn(0x7fe2d80f7190 msgr2=0x7fe2d80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:56.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.226+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 shutdown_connections 2023-12-19T18:45:56.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:56.226+0000 7fe2dcf57700 1 -- 172.21.15.59:0/4195745283 wait complete. 2023-12-19T18:45:56.375 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:56 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/4195745283' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2023-12-19T18:45:56.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:56 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/4195745283' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2023-12-19T18:45:57.008 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T18:45:57.019 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:45:57.019 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2023-12-19T18:45:57.293 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:45:57.413 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/4195745283' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2023-12-19T18:45:57.413 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 1 up:standby 2023-12-19T18:45:57.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:57.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:57.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:57.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.dfrkla", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T18:45:57.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.dfrkla", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T18:45:57.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:57.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: Deploying daemon mds.cephfs.smithi089.dfrkla on smithi089 2023-12-19T18:45:57.414 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:57 smithi059 ceph-mon[80661]: pgmap v178: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2023-12-19T18:45:57.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/4195745283' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2023-12-19T18:45:57.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 1 up:standby 2023-12-19T18:45:57.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:57.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:57.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:57.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.dfrkla", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T18:45:57.505 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi089.dfrkla", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T18:45:57.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:45:57.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: Deploying daemon mds.cephfs.smithi089.dfrkla on smithi089 2023-12-19T18:45:57.506 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:57 smithi089 ceph-mon[88459]: pgmap v178: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2023-12-19T18:45:58.356 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:58 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:boot 2023-12-19T18:45:58.357 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:58 smithi059 ceph-mon[80661]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 2 up:standby 2023-12-19T18:45:58.357 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:58 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.rtqzmc"}]: dispatch 2023-12-19T18:45:58.465 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:58 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:boot 2023-12-19T18:45:58.466 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:58 smithi089 ceph-mon[88459]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 2 up:standby 2023-12-19T18:45:58.466 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:58 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.rtqzmc"}]: dispatch 2023-12-19T18:45:58.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.840+0000 7f4fa1702700 1 -- 172.21.15.59:0/3770257211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 msgr2=0x7f4f9c0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:58.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.840+0000 7f4fa1702700 1 --2- 172.21.15.59:0/3770257211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 0x7f4f9c0fbf00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c004940 tx=0x7f4f8c009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:58.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.840+0000 7f4fa1702700 1 -- 172.21.15.59:0/3770257211 shutdown_connections 2023-12-19T18:45:58.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.840+0000 7f4fa1702700 1 --2- 172.21.15.59:0/3770257211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f9c0fcd00 0x7f4f9c0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:58.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.840+0000 7f4fa1702700 1 --2- 172.21.15.59:0/3770257211 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 0x7f4f9c0fbf00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:58.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.840+0000 7f4fa1702700 1 -- 172.21.15.59:0/3770257211 >> 172.21.15.59:0/3770257211 conn(0x7f4f9c0f71d0 msgr2=0x7f4f9c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.840+0000 7f4fa1702700 1 -- 172.21.15.59:0/3770257211 shutdown_connections 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.840+0000 7f4fa1702700 1 -- 172.21.15.59:0/3770257211 wait complete. 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.840+0000 7f4fa1702700 1 Processor -- start 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4fa1702700 1 -- start start 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4fa1702700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 0x7f4f9c10c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4fa1702700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f9c0fcd00 0x7f4f9c10c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4fa1702700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c10cf10 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4fa1702700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9c10d7b0 con 0x7f4f9c0fcd00 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f9affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 0x7f4f9c10c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f9affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 0x7f4f9c10c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60310/0 (socket says 172.21.15.59:60310) 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f9affd700 1 -- 172.21.15.59:0/733843582 learned_addr learned my addr 172.21.15.59:0/733843582 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f9a7fc700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f9c0fcd00 0x7f4f9c10c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f9affd700 1 -- 172.21.15.59:0/733843582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f9c0fcd00 msgr2=0x7f4f9c10c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:58.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f9affd700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f9c0fcd00 0x7f4f9c10c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f9affd700 1 -- 172.21.15.59:0/733843582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c005040 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f9a7fc700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f9c0fcd00 0x7f4f9c10c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f9affd700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 0x7f4f9c10c3d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009380 tx=0x7f4f8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f837fe700 1 -- 172.21.15.59:0/733843582 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c00e040 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f837fe700 1 -- 172.21.15.59:0/733843582 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f8c01e8d0 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9c10da30 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.841+0000 7f4f837fe700 1 -- 172.21.15.59:0/733843582 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c0136d0 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.842+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9c10df10 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.843+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9c04bb80 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.843+0000 7f4f837fe700 1 -- 172.21.15.59:0/733843582 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4f8c01e420 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.843+0000 7f4f837fe700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f84068f60 0x7f4f8406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.844+0000 7f4f9a7fc700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f84068f60 0x7f4f8406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.844+0000 7f4f837fe700 1 -- 172.21.15.59:0/733843582 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4f8c08ae30 con 0x7f4f9c0fbaf0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.844+0000 7f4f9a7fc700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f84068f60 0x7f4f8406b410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009b80 tx=0x7f4f9000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:45:58.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:58.848+0000 7f4f837fe700 1 -- 172.21.15.59:0/733843582 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4f8c05ad10 con 0x7f4f9c0fbaf0 2023-12-19T18:45:59.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.114+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4f9c104490 con 0x7f4f9c0fbaf0 2023-12-19T18:45:59.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.596+0000 7f4f837fe700 1 -- 172.21.15.59:0/733843582 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7f4f8c05a8a0 con 0x7f4f9c0fbaf0 2023-12-19T18:45:59.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f84068f60 msgr2=0x7f4f8406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:59.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f84068f60 0x7f4f8406b410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009b80 tx=0x7f4f9000b040 comp rx=0 tx=0).stop 2023-12-19T18:45:59.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 msgr2=0x7f4f9c10c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:45:59.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 0x7f4f9c10c3d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009380 tx=0x7f4f8c009e30 comp rx=0 tx=0).stop 2023-12-19T18:45:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 shutdown_connections 2023-12-19T18:45:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f84068f60 0x7f4f8406b410 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f9c0fbaf0 0x7f4f9c10c3d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 --2- 172.21.15.59:0/733843582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f9c0fcd00 0x7f4f9c10c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:45:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 >> 172.21.15.59:0/733843582 conn(0x7f4f9c0f71d0 msgr2=0x7f4f9c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:45:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 shutdown_connections 2023-12-19T18:45:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:45:59.598+0000 7f4fa1702700 1 -- 172.21.15.59:0/733843582 wait complete. 2023-12-19T18:45:59.600 INFO:teuthology.orchestra.run.smithi059.stderr:inline data enabled 2023-12-19T18:45:59.725 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:59 smithi059 ceph-mon[80661]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2023-12-19T18:45:59.725 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.726 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.726 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.726 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.726 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.726 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.726 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:59 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:45:59.726 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:45:59 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/733843582' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2023-12-19T18:45:59.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:59 smithi089 ceph-mon[88459]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2023-12-19T18:45:59.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:45:59.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:59 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:45:59.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:45:59 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/733843582' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2023-12-19T18:46:00.682 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T18:46:00.693 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:46:00.694 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph fs dump' 2023-12-19T18:46:00.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:00 smithi059 ceph-mon[80661]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2023-12-19T18:46:00.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:00 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.89:6826/3691364030,v1:172.21.15.89:6827/3691364030] up:boot 2023-12-19T18:46:00.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:00 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/733843582' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2023-12-19T18:46:00.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:00 smithi059 ceph-mon[80661]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 3 up:standby 2023-12-19T18:46:00.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:00 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.dfrkla"}]: dispatch 2023-12-19T18:46:01.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:00 smithi089 ceph-mon[88459]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2023-12-19T18:46:01.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:00 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.89:6826/3691364030,v1:172.21.15.89:6827/3691364030] up:boot 2023-12-19T18:46:01.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:00 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/733843582' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2023-12-19T18:46:01.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:00 smithi089 ceph-mon[88459]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 3 up:standby 2023-12-19T18:46:01.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:00 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi089.dfrkla"}]: dispatch 2023-12-19T18:46:01.927 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:01.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2023-12-19T18:46:01.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.89:6824/1622105230,v1:172.21.15.89:6825/1622105230] up:standby 2023-12-19T18:46:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:standby 2023-12-19T18:46:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: Dropping low affinity active daemon mds.cephfs.smithi059.abqklt in favor of higher affinity standby. 2023-12-19T18:46:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: Replacing daemon mds.cephfs.smithi059.abqklt as rank 0 with standby daemon mds.cephfs.smithi059.rtqzmc 2023-12-19T18:46:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-12-19T18:46:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 3 up:standby 2023-12-19T18:46:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: osdmap e47: 6 total, 6 up, 6 in 2023-12-19T18:46:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: fsmap cephfs:1/1 {0=cephfs.smithi059.rtqzmc=up:replay} 2 up:standby 2023-12-19T18:46:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:01 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2023-12-19T18:46:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.89:6824/1622105230,v1:172.21.15.89:6825/1622105230] up:standby 2023-12-19T18:46:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:standby 2023-12-19T18:46:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: Dropping low affinity active daemon mds.cephfs.smithi059.abqklt in favor of higher affinity standby. 2023-12-19T18:46:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: Replacing daemon mds.cephfs.smithi059.abqklt as rank 0 with standby daemon mds.cephfs.smithi059.rtqzmc 2023-12-19T18:46:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-12-19T18:46:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: fsmap cephfs:1 {0=cephfs.smithi059.abqklt=up:active} 3 up:standby 2023-12-19T18:46:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: osdmap e47: 6 total, 6 up, 6 in 2023-12-19T18:46:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: fsmap cephfs:1/1 {0=cephfs.smithi059.rtqzmc=up:replay} 2 up:standby 2023-12-19T18:46:02.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:02.006 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:01 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:02.701 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:02 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:reconnect 2023-12-19T18:46:02.702 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:02 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.59:6826/1872103080,v1:172.21.15.59:6827/1872103080] up:boot 2023-12-19T18:46:02.702 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:02 smithi059 ceph-mon[80661]: fsmap cephfs:1/1 {0=cephfs.smithi059.rtqzmc=up:reconnect} 3 up:standby 2023-12-19T18:46:02.702 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:02 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.abqklt"}]: dispatch 2023-12-19T18:46:03.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:02 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:reconnect 2023-12-19T18:46:03.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:02 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.59:6826/1872103080,v1:172.21.15.59:6827/1872103080] up:boot 2023-12-19T18:46:03.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:02 smithi089 ceph-mon[88459]: fsmap cephfs:1/1 {0=cephfs.smithi059.rtqzmc=up:reconnect} 3 up:standby 2023-12-19T18:46:03.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:02 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi059.abqklt"}]: dispatch 2023-12-19T18:46:03.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:03 smithi059 ceph-mon[80661]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 204 B/s wr, 1 op/s 2023-12-19T18:46:03.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:03 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:rejoin 2023-12-19T18:46:03.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:03 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.89:6826/3691364030,v1:172.21.15.89:6827/3691364030] up:standby 2023-12-19T18:46:03.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:03 smithi059 ceph-mon[80661]: fsmap cephfs:1/1 {0=cephfs.smithi059.rtqzmc=up:rejoin} 3 up:standby 2023-12-19T18:46:03.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:03 smithi059 ceph-mon[80661]: daemon mds.cephfs.smithi059.rtqzmc is now active in filesystem cephfs as rank 0 2023-12-19T18:46:03.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:03 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:03.932 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:03 smithi089 ceph-mon[88459]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 614 B/s rd, 204 B/s wr, 1 op/s 2023-12-19T18:46:03.932 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:03 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:rejoin 2023-12-19T18:46:03.932 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:03 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.89:6826/3691364030,v1:172.21.15.89:6827/3691364030] up:standby 2023-12-19T18:46:03.932 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:03 smithi089 ceph-mon[88459]: fsmap cephfs:1/1 {0=cephfs.smithi059.rtqzmc=up:rejoin} 3 up:standby 2023-12-19T18:46:03.932 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:03 smithi089 ceph-mon[88459]: daemon mds.cephfs.smithi059.rtqzmc is now active in filesystem cephfs as rank 0 2023-12-19T18:46:03.933 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:03 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:04.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:04 smithi059 ceph-mon[80661]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-12-19T18:46:04.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:04 smithi059 ceph-mon[80661]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:active 2023-12-19T18:46:04.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:04 smithi059 ceph-mon[80661]: fsmap cephfs:1 {0=cephfs.smithi059.rtqzmc=up:active} 3 up:standby 2023-12-19T18:46:04.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.747+0000 7fa291b8b700 1 -- 172.21.15.59:0/3244482740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 msgr2=0x7fa28c0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:04.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.747+0000 7fa291b8b700 1 --2- 172.21.15.59:0/3244482740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 0x7fa28c0fe250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa280004940 tx=0x7fa280009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:04.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 -- 172.21.15.59:0/3244482740 shutdown_connections 2023-12-19T18:46:04.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 --2- 172.21.15.59:0/3244482740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 0x7fa28c0fe250 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 --2- 172.21.15.59:0/3244482740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa28c0fcbf0 0x7fa28c0fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 -- 172.21.15.59:0/3244482740 >> 172.21.15.59:0/3244482740 conn(0x7fa28c0f82d0 msgr2=0x7fa28c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 -- 172.21.15.59:0/3244482740 shutdown_connections 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 -- 172.21.15.59:0/3244482740 wait complete. 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 Processor -- start 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 -- start start 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa28c0fcbf0 0x7fa28c071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 0x7fa28c06ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c070590 con 0x7fa28c0fde00 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.748+0000 7fa291b8b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c070700 con 0x7fa28c0fcbf0 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa28affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 0x7fa28c06ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:04.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa28affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 0x7fa28c06ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41968/0 (socket says 172.21.15.59:41968) 2023-12-19T18:46:04.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa28affd700 1 -- 172.21.15.59:0/2199536648 learned_addr learned my addr 172.21.15.59:0/2199536648 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:04.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa28b7fe700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa28c0fcbf0 0x7fa28c071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa28affd700 1 -- 172.21.15.59:0/2199536648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa28c0fcbf0 msgr2=0x7fa28c071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa28affd700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa28c0fcbf0 0x7fa28c071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa28affd700 1 -- 172.21.15.59:0/2199536648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280005040 con 0x7fa28c0fde00 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa28affd700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 0x7fa28c06ff90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa280009fd0 tx=0x7fa280009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa273fff700 1 -- 172.21.15.59:0/2199536648 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28000e040 con 0x7fa28c0fde00 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa273fff700 1 -- 172.21.15.59:0/2199536648 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa280013420 con 0x7fa28c0fde00 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c070980 con 0x7fa28c0fde00 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c115610 con 0x7fa28c0fde00 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.749+0000 7fa273fff700 1 -- 172.21.15.59:0/2199536648 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28001c610 con 0x7fa28c0fde00 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.750+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28c04bb80 con 0x7fa28c0fde00 2023-12-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.751+0000 7fa273fff700 1 -- 172.21.15.59:0/2199536648 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa280013590 con 0x7fa28c0fde00 2023-12-19T18:46:04.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.751+0000 7fa273fff700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa274068f70 0x7fa27406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:04.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.751+0000 7fa273fff700 1 -- 172.21.15.59:0/2199536648 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa2800880b0 con 0x7fa28c0fde00 2023-12-19T18:46:04.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.751+0000 7fa28b7fe700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa274068f70 0x7fa27406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:04.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.752+0000 7fa28b7fe700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa274068f70 0x7fa27406b420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa28c0718a0 tx=0x7fa27c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:04.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:04 smithi089 ceph-mon[88459]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-12-19T18:46:04.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:04 smithi089 ceph-mon[88459]: mds.? [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] up:active 2023-12-19T18:46:04.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:04 smithi089 ceph-mon[88459]: fsmap cephfs:1 {0=cephfs.smithi059.rtqzmc=up:active} 3 up:standby 2023-12-19T18:46:04.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.756+0000 7fa273fff700 1 -- 172.21.15.59:0/2199536648 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa280057fc0 con 0x7fa28c0fde00 2023-12-19T18:46:04.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.999+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa28c1159f0 con 0x7fa28c0fde00 2023-12-19T18:46:05.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:04.999+0000 7fa273fff700 1 -- 172.21.15.59:0/2199536648 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1805 (secure 0 0 0) 0x7fa280057b50 con 0x7fa28c0fde00 2023-12-19T18:46:05.000 INFO:teuthology.orchestra.run.smithi059.stdout:e15 2023-12-19T18:46:05.000 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-19T18:46:05.000 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:46:05.000 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2023-12-19T18:46:05.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 15 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:created 2023-12-19T18:45:45.631056+0000 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2023-12-19T18:46:03.615908+0000 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2023-12-19T18:46:05.001 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 47 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:in 0 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:up {0=14474} 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data enabled 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 1 2023-12-19T18:46:05.002 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.rtqzmc{0:14474} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:46:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:05.003 INFO:teuthology.orchestra.run.smithi059.stdout:Standby daemons: 2023-12-19T18:46:05.003 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:05.003 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.abqklt{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.59:6826/1872103080,v1:172.21.15.59:6827/1872103080] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:46:05.003 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi089.dpgsbt{-1:24291} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.89:6824/1622105230,v1:172.21.15.89:6825/1622105230] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:46:05.003 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi089.dfrkla{-1:24301} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.89:6826/3691364030,v1:172.21.15.89:6827/3691364030] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa274068f70 msgr2=0x7fa27406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa274068f70 0x7fa27406b420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa28c0718a0 tx=0x7fa27c00b040 comp rx=0 tx=0).stop 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 msgr2=0x7fa28c06ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 0x7fa28c06ff90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa280009fd0 tx=0x7fa280009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 shutdown_connections 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa28c0fcbf0 0x7fa28c071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fa274068f70 0x7fa27406b420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 --2- 172.21.15.59:0/2199536648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa28c0fde00 0x7fa28c06ff90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 >> 172.21.15.59:0/2199536648 conn(0x7fa28c0f82d0 msgr2=0x7fa28c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:05.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 shutdown_connections 2023-12-19T18:46:05.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:05.002+0000 7fa291b8b700 1 -- 172.21.15.59:0/2199536648 wait complete. 2023-12-19T18:46:05.005 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 15 2023-12-19T18:46:05.676 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:05 smithi059 ceph-mon[80661]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-12-19T18:46:05.677 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:05 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2199536648' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:46:05.702 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2023-12-19T18:46:06.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:05 smithi089 ceph-mon[88459]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-12-19T18:46:06.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:05 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/2199536648' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:46:06.005 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:07.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:07.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:07.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:46:07.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:46:07.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:07.311 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:46:07.312 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:07 smithi059 ceph-mon[80661]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 3 op/s 2023-12-19T18:46:07.376 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:07.376 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:07.376 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:46:07.376 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:46:07.376 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:07.376 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:46:07.376 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:07 smithi089 ceph-mon[88459]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 3 op/s 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 -- 172.21.15.59:0/507724550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 msgr2=0x7f40a00fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 --2- 172.21.15.59:0/507724550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 0x7f40a00fd000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4090004940 tx=0x7f4090009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 -- 172.21.15.59:0/507724550 shutdown_connections 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 --2- 172.21.15.59:0/507724550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a00fde00 0x7f40a00fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 --2- 172.21.15.59:0/507724550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 0x7f40a00fd000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 -- 172.21.15.59:0/507724550 >> 172.21.15.59:0/507724550 conn(0x7f40a00f82d0 msgr2=0x7f40a00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 -- 172.21.15.59:0/507724550 shutdown_connections 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 -- 172.21.15.59:0/507724550 wait complete. 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 Processor -- start 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.522+0000 7f40a4ed3700 1 -- start start 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f40a4ed3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 0x7f40a0071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f40a4ed3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a00fde00 0x7f40a006ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:07.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f40a4ed3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0070590 con 0x7f40a00fcbf0 2023-12-19T18:46:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f40a4ed3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a0070700 con 0x7f40a00fde00 2023-12-19T18:46:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f409e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 0x7f40a0071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f409e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 0x7f40a0071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41980/0 (socket says 172.21.15.59:41980) 2023-12-19T18:46:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f409e59c700 1 -- 172.21.15.59:0/930876047 learned_addr learned my addr 172.21.15.59:0/930876047 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f409dd9b700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a00fde00 0x7f40a006ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f409e59c700 1 -- 172.21.15.59:0/930876047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a00fde00 msgr2=0x7f40a006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f409e59c700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a00fde00 0x7f40a006ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:07.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f409e59c700 1 -- 172.21.15.59:0/930876047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4090005040 con 0x7f40a00fcbf0 2023-12-19T18:46:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f409e59c700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 0x7f40a0071b10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4090009360 tx=0x7f4090009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f408effd700 1 -- 172.21.15.59:0/930876047 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f409000e040 con 0x7f40a00fcbf0 2023-12-19T18:46:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.523+0000 7f408effd700 1 -- 172.21.15.59:0/930876047 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4090004290 con 0x7f40a00fcbf0 2023-12-19T18:46:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.524+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a0070980 con 0x7f40a00fcbf0 2023-12-19T18:46:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.524+0000 7f408effd700 1 -- 172.21.15.59:0/930876047 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f409001c3f0 con 0x7f40a00fcbf0 2023-12-19T18:46:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.524+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a0115520 con 0x7f40a00fcbf0 2023-12-19T18:46:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.525+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a004bb80 con 0x7f40a00fcbf0 2023-12-19T18:46:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.525+0000 7f408effd700 1 -- 172.21.15.59:0/930876047 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f40900137e0 con 0x7f40a00fcbf0 2023-12-19T18:46:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.525+0000 7f408effd700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4088068f70 0x7f408806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.525+0000 7f408effd700 1 -- 172.21.15.59:0/930876047 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4090088120 con 0x7f40a00fcbf0 2023-12-19T18:46:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.525+0000 7f409dd9b700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4088068f70 0x7f408806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.526+0000 7f409dd9b700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4088068f70 0x7f408806b420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4094004f40 tx=0x7f409400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:07.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.530+0000 7f408effd700 1 -- 172.21.15.59:0/930876047 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4090057fb0 con 0x7f40a00fcbf0 2023-12-19T18:46:07.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.772+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40a0105590 con 0x7f40a00fcbf0 2023-12-19T18:46:07.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.773+0000 7f408effd700 1 -- 172.21.15.59:0/930876047 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4685 (secure 0 0 0) 0x7f409005d030 con 0x7f40a00fcbf0 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4088068f70 msgr2=0x7f408806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4088068f70 0x7f408806b420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4094004f40 tx=0x7f409400b040 comp rx=0 tx=0).stop 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 msgr2=0x7f40a0071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 0x7f40a0071b10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4090009360 tx=0x7f4090009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 shutdown_connections 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4088068f70 0x7f408806b420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40a00fcbf0 0x7f40a0071b10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 --2- 172.21.15.59:0/930876047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f40a00fde00 0x7f40a006ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 >> 172.21.15.59:0/930876047 conn(0x7f40a00f82d0 msgr2=0x7f40a0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 shutdown_connections 2023-12-19T18:46:07.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:07.775+0000 7f40a4ed3700 1 -- 172.21.15.59:0/930876047 wait complete. 2023-12-19T18:46:07.777 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 15 2023-12-19T18:46:07.786 INFO:teuthology.orchestra.run.smithi059.stdout:true 2023-12-19T18:46:08.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:08 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/930876047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-19T18:46:08.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:08 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/930876047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-19T18:46:08.492 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2023-12-19T18:46:08.831 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:09.247 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:09.247 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:09.247 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:09 smithi059 ceph-mon[80661]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 4 op/s 2023-12-19T18:46:09.247 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:09.247 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:09 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:46:09.415 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:09.415 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:09.415 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:09 smithi089 ceph-mon[88459]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 4 op/s 2023-12-19T18:46:09.415 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:09.415 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:09 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:46:10.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.398+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2460385880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fcd90 msgr2=0x7f15dc0fd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:10.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.398+0000 7f15e2f4a700 1 --2- 172.21.15.59:0/2460385880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fcd90 0x7f15dc0fd200 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f15c8004940 tx=0x7f15c8009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:10.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.398+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2460385880 shutdown_connections 2023-12-19T18:46:10.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.398+0000 7f15e2f4a700 1 --2- 172.21.15.59:0/2460385880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fcd90 0x7f15dc0fd200 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:10.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.398+0000 7f15e2f4a700 1 --2- 172.21.15.59:0/2460385880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15dc0fbae0 0x7f15dc0fbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:10.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.398+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2460385880 >> 172.21.15.59:0/2460385880 conn(0x7f15dc0f7240 msgr2=0x7f15dc0f9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:10.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.398+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2460385880 shutdown_connections 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.398+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2460385880 wait complete. 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.399+0000 7f15e2f4a700 1 Processor -- start 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.399+0000 7f15e2f4a700 1 -- start start 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.399+0000 7f15e2f4a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fbae0 0x7f15dc0ffc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.399+0000 7f15e2f4a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15dc0fcd90 0x7f15dc0fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.399+0000 7f15e2f4a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0fe760 con 0x7f15dc0fbae0 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.399+0000 7f15e2f4a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0fe8d0 con 0x7f15dc0fcd90 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.399+0000 7f15e0ce6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fbae0 0x7f15dc0ffc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.399+0000 7f15e0ce6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fbae0 0x7f15dc0ffc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41996/0 (socket says 172.21.15.59:41996) 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.399+0000 7f15e0ce6700 1 -- 172.21.15.59:0/2995888469 learned_addr learned my addr 172.21.15.59:0/2995888469 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15dbfff700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15dc0fcd90 0x7f15dc0fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15e0ce6700 1 -- 172.21.15.59:0/2995888469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15dc0fcd90 msgr2=0x7f15dc0fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15e0ce6700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15dc0fcd90 0x7f15dc0fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15e0ce6700 1 -- 172.21.15.59:0/2995888469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8005040 con 0x7f15dc0fbae0 2023-12-19T18:46:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15dbfff700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15dc0fcd90 0x7f15dc0fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T18:46:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15e0ce6700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fbae0 0x7f15dc0ffc50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f15d000b790 tx=0x7f15d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15d97fa700 1 -- 172.21.15.59:0/2995888469 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15d000f770 con 0x7f15dc0fbae0 2023-12-19T18:46:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15d97fa700 1 -- 172.21.15.59:0/2995888469 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15d000fd80 con 0x7f15dc0fbae0 2023-12-19T18:46:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15dc0febb0 con 0x7f15dc0fbae0 2023-12-19T18:46:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15d97fa700 1 -- 172.21.15.59:0/2995888469 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15d000d310 con 0x7f15dc0fbae0 2023-12-19T18:46:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.400+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15dc0fef30 con 0x7f15dc0fbae0 2023-12-19T18:46:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.401+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15dc04bb80 con 0x7f15dc0fbae0 2023-12-19T18:46:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.402+0000 7f15d97fa700 1 -- 172.21.15.59:0/2995888469 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f15d0010420 con 0x7f15dc0fbae0 2023-12-19T18:46:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.402+0000 7f15d97fa700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f15cc068f60 0x7f15cc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.402+0000 7f15d97fa700 1 -- 172.21.15.59:0/2995888469 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f15d0086dd0 con 0x7f15dc0fbae0 2023-12-19T18:46:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.402+0000 7f15dbfff700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f15cc068f60 0x7f15cc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.403+0000 7f15dbfff700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f15cc068f60 0x7f15cc06b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f15c8004560 tx=0x7f15c8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.407+0000 7f15d97fa700 1 -- 172.21.15.59:0/2995888469 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f15d0056ce0 con 0x7f15dc0fbae0 2023-12-19T18:46:10.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.691+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f15dc104510 con 0x7f15dc0fbae0 2023-12-19T18:46:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.692+0000 7f15d97fa700 1 -- 172.21.15.59:0/2995888469 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+99 (secure 0 0 0) 0x7f15dc104510 con 0x7f15dc0fbae0 2023-12-19T18:46:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f15cc068f60 msgr2=0x7f15cc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f15cc068f60 0x7f15cc06b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f15c8004560 tx=0x7f15c8009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fbae0 msgr2=0x7f15dc0ffc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fbae0 0x7f15dc0ffc50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f15d000b790 tx=0x7f15d0009250 comp rx=0 tx=0).stop 2023-12-19T18:46:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 shutdown_connections 2023-12-19T18:46:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f15cc068f60 0x7f15cc06b410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:10.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15dc0fbae0 0x7f15dc0ffc50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:10.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 --2- 172.21.15.59:0/2995888469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15dc0fcd90 0x7f15dc0fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:10.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 >> 172.21.15.59:0/2995888469 conn(0x7f15dc0f7240 msgr2=0x7f15dc065c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:10.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 shutdown_connections 2023-12-19T18:46:10.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:10.694+0000 7f15e2f4a700 1 -- 172.21.15.59:0/2995888469 wait complete. 2023-12-19T18:46:10.706 INFO:teuthology.orchestra.run.smithi059.stdout:true 2023-12-19T18:46:11.654 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2023-12-19T18:46:11.684 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T18:46:11.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.801+0000 7f3677007640 1 -- 172.21.15.59:0/4227193011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 msgr2=0x7f36700fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:11.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.801+0000 7f3677007640 1 --2- 172.21.15.59:0/4227193011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 0x7f36700fb2f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f36600048d0 tx=0x7f3660009da0 comp rx=0 tx=0).stop 2023-12-19T18:46:11.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.801+0000 7f3677007640 1 -- 172.21.15.59:0/4227193011 shutdown_connections 2023-12-19T18:46:11.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.801+0000 7f3677007640 1 --2- 172.21.15.59:0/4227193011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36700fc0d0 0x7f36700fc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:11.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.801+0000 7f3677007640 1 --2- 172.21.15.59:0/4227193011 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 0x7f36700fb2f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:11.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.801+0000 7f3677007640 1 -- 172.21.15.59:0/4227193011 >> 172.21.15.59:0/4227193011 conn(0x7f36700f6650 msgr2=0x7f36700f8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:11.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3677007640 1 -- 172.21.15.59:0/4227193011 shutdown_connections 2023-12-19T18:46:11.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3677007640 1 -- 172.21.15.59:0/4227193011 wait complete. 2023-12-19T18:46:11.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3677007640 1 Processor -- start 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3677007640 1 -- start start 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3677007640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 0x7f367010c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3677007640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36700fc0d0 0x7f367010ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3677007640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367010d310 con 0x7f36700faec0 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3677007640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670111d60 con 0x7f36700fc0d0 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3674da3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 0x7f367010c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3674da3640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 0x7f367010c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35896/0 (socket says 172.21.15.59:35896) 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.802+0000 7f3674da3640 1 -- 172.21.15.59:0/3796773394 learned_addr learned my addr 172.21.15.59:0/3796773394 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f366ffff640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36700fc0d0 0x7f367010ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f3674da3640 1 -- 172.21.15.59:0/3796773394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36700fc0d0 msgr2=0x7f367010ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f3674da3640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36700fc0d0 0x7f367010ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:11.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f3674da3640 1 -- 172.21.15.59:0/3796773394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3660005040 con 0x7f36700faec0 2023-12-19T18:46:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f366ffff640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36700fc0d0 0x7f367010ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T18:46:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f3674da3640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 0x7f367010c780 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3660034040 tx=0x7f3660009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f366d7fa640 1 -- 172.21.15.59:0/3796773394 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36600094c0 con 0x7f36700faec0 2023-12-19T18:46:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f366d7fa640 1 -- 172.21.15.59:0/3796773394 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3660009620 con 0x7f36700faec0 2023-12-19T18:46:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f366d7fa640 1 -- 172.21.15.59:0/3796773394 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f366003b400 con 0x7f36700faec0 2023-12-19T18:46:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3670111ea0 con 0x7f36700faec0 2023-12-19T18:46:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.803+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36701123b0 con 0x7f36700faec0 2023-12-19T18:46:11.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.804+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36701038b0 con 0x7f36700faec0 2023-12-19T18:46:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.804+0000 7f366d7fa640 1 -- 172.21.15.59:0/3796773394 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f366002f070 con 0x7f36700faec0 2023-12-19T18:46:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.805+0000 7f366d7fa640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3658068ed0 0x7f365806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.805+0000 7f366d7fa640 1 -- 172.21.15.59:0/3796773394 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f36600a92d0 con 0x7f36700faec0 2023-12-19T18:46:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.805+0000 7f366ffff640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3658068ed0 0x7f365806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:11.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.806+0000 7f366ffff640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3658068ed0 0x7f365806b380 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f367010dcd0 tx=0x7f366400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:11.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.810+0000 7f366d7fa640 1 -- 172.21.15.59:0/3796773394 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3660075800 con 0x7f36700faec0 2023-12-19T18:46:11.896 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:11 smithi059 ceph-mon[80661]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2023-12-19T18:46:11.896 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:11 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2995888469' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2023-12-19T18:46:11.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.992+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3670112680 con 0x7f36700faec0 2023-12-19T18:46:11.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.993+0000 7f366d7fa640 1 -- 172.21.15.59:0/3796773394 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14343 (secure 0 0 0) 0x7f3660009790 con 0x7f36700faec0 2023-12-19T18:46:11.994 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:11.994 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":47,"fsid":"bf854274-9e9d-11ee-95a5-87774f69a715","created":"2023-12-19T18:38:57.167145+0000","modified":"2023-12-19T18:46:00.601240+0000","last_up_change":"2023-12-19T18:44:30.032151+0000","last_in_change":"2023-12-19T18:44:11.480055+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T18:43:21.014480+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2023-12-19T18:45:44.455944+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2023-12-19T18:45:45.059947+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a33e1310-4169-41e4-bd1f-748898ef9289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6803","nonce":177591080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6805","nonce":177591080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6809","nonce":177591080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":177591080},{"type":"v1","addr":"172.21.15.59:6807","nonce":177591080}]},"public_addr":"172.21.15.59:6803/177591080","cluster_addr":"172.21.15.59:6805/177591080","heartbeat_back_addr":"172.21.15.59:6809/177591080","heartbeat_front_addr":"172.21.15.59:6807/177591080","state":["exists","up"]},{"osd":1,"uuid":"73a2cb1c-90fe-4d56-adfd-78a688c6b2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6811","nonce":1239732570}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6813","nonce":1239732570}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6816","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6817","nonce":1239732570}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":1239732570},{"type":"v1","addr":"172.21.15.59:6815","nonce":1239732570}]},"public_addr":"172.21.15.59:6811/1239732570","cluster_addr":"172.21.15.59:6813/1239732570","heartbeat_back_addr":"172.21.15.59:6817/1239732570","heartbeat_front_addr":"172.21.15.59:6815/1239732570","state":["exists","up"]},{"osd":2,"uuid":"0003bcc2-1e16-44d8-a031-8e26fa6c7467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6818","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6819","nonce":240742436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6820","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6821","nonce":240742436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6824","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6825","nonce":240742436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6822","nonce":240742436},{"type":"v1","addr":"172.21.15.59:6823","nonce":240742436}]},"public_addr":"172.21.15.59:6819/240742436","cluster_addr":"172.21.15.59:6821/240742436","heartbeat_back_addr":"172.21.15.59:6825/240742436","heartbeat_front_addr":"172.21.15.59:6823/240742436","state":["exists","up"]},{"osd":3,"uuid":"08b40954-a1b3-4754-8494-e3a5423406c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6801","nonce":2655128462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6803","nonce":2655128462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6807","nonce":2655128462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2655128462},{"type":"v1","addr":"172.21.15.89:6805","nonce":2655128462}]},"public_addr":"172.21.15.89:6801/2655128462","cluster_addr":"172.21.15.89:6803/2655128462","heartbeat_back_addr":"172.21.15.89:6807/2655128462","heartbeat_front_addr":"172.21.15.89:6805/2655128462","state":["exists","up"]},{"osd":4,"uuid":"152f485f-9a72-40e5-9a4d-535e4caf73ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6809","nonce":2501804384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6811","nonce":2501804384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6815","nonce":2501804384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2501804384},{"type":"v1","addr":"172.21.15.89:6813","nonce":2501804384}]},"public_addr":"172.21.15.89:6809/2501804384","cluster_addr":"172.21.15.89:6811/2501804384","heartbeat_back_addr":"172.21.15.89:6815/2501804384","heartbeat_front_addr":"172.21.15.89:6813/2501804384","state":["exists","up"]},{"osd":5,"uuid":"a92c1e7d-9b7c-4594-b8db-7f812da2afc6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6817","nonce":1872882017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6819","nonce":1872882017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6823","nonce":1872882017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":1872882017},{"type":"v1","addr":"172.21.15.89:6821","nonce":1872882017}]},"public_addr":"172.21.15.89:6817/1872882017","cluster_addr":"172.21.15.89:6819/1872882017","heartbeat_back_addr":"172.21.15.89:6823/1872882017","heartbeat_front_addr":"172.21.15.89:6821/1872882017","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:42:23.049328+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:42:50.671474+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:43:17.816383+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:43:40.185718+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:44:04.182008+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T18:44:28.308577+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.59:6827/663831313":"2023-12-20T18:46:00.601207+0000","172.21.15.59:6800/2067989811":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/1707232246":"2023-12-20T18:39:27.397002+0000","172.21.15.59:6801/1973265269":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/1665894963":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/990606911":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/3060878312":"2023-12-20T18:39:27.397002+0000","172.21.15.59:0/1003305302":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/4127123819":"2023-12-20T18:41:08.486998+0000","172.21.15.59:6800/1973265269":"2023-12-20T18:39:27.397002+0000","172.21.15.59:6801/2067989811":"2023-12-20T18:39:58.643904+0000","172.21.15.59:6826/663831313":"2023-12-20T18:46:00.601207+0000","172.21.15.59:6801/6145510":"2023-12-20T18:41:08.486998+0000","172.21.15.59:6800/6145510":"2023-12-20T18:41:08.486998+0000","172.21.15.59:0/3575411648":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/3905471834":"2023-12-20T18:39:58.643904+0000","172.21.15.59:0/3325232385":"2023-12-20T18:41:08.486998+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.994+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3658068ed0 msgr2=0x7f365806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.994+0000 7f3677007640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3658068ed0 0x7f365806b380 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f367010dcd0 tx=0x7f366400b040 comp rx=0 tx=0).stop 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.994+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 msgr2=0x7f367010c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.994+0000 7f3677007640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 0x7f367010c780 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3660034040 tx=0x7f3660009da0 comp rx=0 tx=0).stop 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.995+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 shutdown_connections 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.995+0000 7f3677007640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3658068ed0 0x7f365806b380 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.995+0000 7f3677007640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36700fc0d0 0x7f367010ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.995+0000 7f3677007640 1 --2- 172.21.15.59:0/3796773394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36700faec0 0x7f367010c780 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.995+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 >> 172.21.15.59:0/3796773394 conn(0x7f36700f6650 msgr2=0x7f36700653a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:11.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.995+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 shutdown_connections 2023-12-19T18:46:11.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:11.995+0000 7f3677007640 1 -- 172.21.15.59:0/3796773394 wait complete. 2023-12-19T18:46:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:11 smithi089 ceph-mon[88459]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2023-12-19T18:46:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:11 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/2995888469' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2023-12-19T18:46:12.006 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T18:43:21.014480+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2023-12-19T18:45:44.455944+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2023-12-19T18:45:45.059947+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T18:46:12.007 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T18:46:12.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.114+0000 7f75ef26e640 1 -- 172.21.15.59:0/2011132048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e8065fc0 msgr2=0x7f75e80feba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.114+0000 7f75ef26e640 1 --2- 172.21.15.59:0/2011132048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e8065fc0 0x7f75e80feba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f75d80048d0 tx=0x7f75d8009da0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.114+0000 7f75ef26e640 1 -- 172.21.15.59:0/2011132048 shutdown_connections 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.114+0000 7f75ef26e640 1 --2- 172.21.15.59:0/2011132048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e80ff0e0 0x7f75e81014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.114+0000 7f75ef26e640 1 --2- 172.21.15.59:0/2011132048 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e8065fc0 0x7f75e80feba0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.114+0000 7f75ef26e640 1 -- 172.21.15.59:0/2011132048 >> 172.21.15.59:0/2011132048 conn(0x7f75e80f6610 msgr2=0x7f75e80f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.114+0000 7f75ef26e640 1 -- 172.21.15.59:0/2011132048 shutdown_connections 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.114+0000 7f75ef26e640 1 -- 172.21.15.59:0/2011132048 wait complete. 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.114+0000 7f75ef26e640 1 Processor -- start 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ef26e640 1 -- start start 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ef26e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e8065fc0 0x7f75e810e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ef26e640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e80ff0e0 0x7f75e810ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:12.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ef26e640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e810f4c0 con 0x7f75e80ff0e0 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ef26e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e8113f10 con 0x7f75e8065fc0 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ec809640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e80ff0e0 0x7f75e810ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ec809640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e80ff0e0 0x7f75e810ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35904/0 (socket says 172.21.15.59:35904) 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ec809640 1 -- 172.21.15.59:0/3328049496 learned_addr learned my addr 172.21.15.59:0/3328049496 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ed00a640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e8065fc0 0x7f75e810e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ec809640 1 -- 172.21.15.59:0/3328049496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e8065fc0 msgr2=0x7f75e810e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ec809640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e8065fc0 0x7f75e810e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ec809640 1 -- 172.21.15.59:0/3328049496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8005040 con 0x7f75e80ff0e0 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ed00a640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e8065fc0 0x7f75e810e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ec809640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e80ff0e0 0x7f75e810ee70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f75dc002a80 tx=0x7f75dc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75e5ffb640 1 -- 172.21.15.59:0/3328049496 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75dc007c30 con 0x7f75e80ff0e0 2023-12-19T18:46:12.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75e5ffb640 1 -- 172.21.15.59:0/3328049496 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75dc00f040 con 0x7f75e80ff0e0 2023-12-19T18:46:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e81140b0 con 0x7f75e80ff0e0 2023-12-19T18:46:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.115+0000 7f75e5ffb640 1 -- 172.21.15.59:0/3328049496 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75dc0095b0 con 0x7f75e80ff0e0 2023-12-19T18:46:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.116+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e81145a0 con 0x7f75e80ff0e0 2023-12-19T18:46:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.117+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e8104950 con 0x7f75e80ff0e0 2023-12-19T18:46:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.117+0000 7f75e5ffb640 1 -- 172.21.15.59:0/3328049496 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f75dc009710 con 0x7f75e80ff0e0 2023-12-19T18:46:12.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.117+0000 7f75e5ffb640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f75d0068ed0 0x7f75d006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:12.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.118+0000 7f75e5ffb640 1 -- 172.21.15.59:0/3328049496 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f75dc0867e0 con 0x7f75e80ff0e0 2023-12-19T18:46:12.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.118+0000 7f75ed00a640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f75d0068ed0 0x7f75d006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:12.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.118+0000 7f75ed00a640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f75d0068ed0 0x7f75d006b380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f75d8007ac0 tx=0x7f75d8009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:12.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.122+0000 7f75e5ffb640 1 -- 172.21.15.59:0/3328049496 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f75dc04ca00 con 0x7f75e80ff0e0 2023-12-19T18:46:12.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.295+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75e8114a40 con 0x7f75e80ff0e0 2023-12-19T18:46:12.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.296+0000 7f75e5ffb640 1 -- 172.21.15.59:0/3328049496 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75dc052be0 con 0x7f75e80ff0e0 2023-12-19T18:46:12.301 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 1 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f75d0068ed0 msgr2=0x7f75d006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f75d0068ed0 0x7f75d006b380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f75d8007ac0 tx=0x7f75d8009da0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e80ff0e0 msgr2=0x7f75e810ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e80ff0e0 0x7f75e810ee70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f75dc002a80 tx=0x7f75dc004600 comp rx=0 tx=0).stop 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 shutdown_connections 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f75d0068ed0 0x7f75d006b380 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75e80ff0e0 0x7f75e810ee70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 --2- 172.21.15.59:0/3328049496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e8065fc0 0x7f75e810e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 >> 172.21.15.59:0/3328049496 conn(0x7f75e80f6610 msgr2=0x7f75e80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 shutdown_connections 2023-12-19T18:46:12.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.298+0000 7f75ef26e640 1 -- 172.21.15.59:0/3328049496 wait complete. 2023-12-19T18:46:12.313 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2023-12-19T18:46:12.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.424+0000 7f6ac7f54640 1 -- 172.21.15.59:0/570444030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac01014b0 msgr2=0x7f6ac0103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.424+0000 7f6ac7f54640 1 --2- 172.21.15.59:0/570444030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac01014b0 0x7f6ac0103890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00048d0 tx=0x7f6ab0009da0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 -- 172.21.15.59:0/570444030 shutdown_connections 2023-12-19T18:46:12.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 --2- 172.21.15.59:0/570444030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac01014b0 0x7f6ac0103890 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 --2- 172.21.15.59:0/570444030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac0064e10 0x7f6ac0100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 -- 172.21.15.59:0/570444030 >> 172.21.15.59:0/570444030 conn(0x7f6ac00f6580 msgr2=0x7f6ac00f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:12.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 -- 172.21.15.59:0/570444030 shutdown_connections 2023-12-19T18:46:12.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 -- 172.21.15.59:0/570444030 wait complete. 2023-12-19T18:46:12.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 Processor -- start 2023-12-19T18:46:12.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 -- start start 2023-12-19T18:46:12.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac0064e10 0x7f6ac010e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:12.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac01014b0 0x7f6ac010ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:12.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac010f490 con 0x7f6ac0064e10 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac7f54640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac0113ee0 con 0x7f6ac01014b0 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac5cf0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac0064e10 0x7f6ac010e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac5cf0640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac0064e10 0x7f6ac010e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35916/0 (socket says 172.21.15.59:35916) 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.425+0000 7f6ac5cf0640 1 -- 172.21.15.59:0/3325918026 learned_addr learned my addr 172.21.15.59:0/3325918026 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ac54ef640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac01014b0 0x7f6ac010ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ac5cf0640 1 -- 172.21.15.59:0/3325918026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac01014b0 msgr2=0x7f6ac010ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ac5cf0640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac01014b0 0x7f6ac010ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ac5cf0640 1 -- 172.21.15.59:0/3325918026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0005040 con 0x7f6ac0064e10 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ac54ef640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac01014b0 0x7f6ac010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ac5cf0640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac0064e10 0x7f6ac010e900 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00eb70 tx=0x7f6abc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ab67fc640 1 -- 172.21.15.59:0/3325918026 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6abc007c30 con 0x7f6ac0064e10 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ab67fc640 1 -- 172.21.15.59:0/3325918026 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6abc00f040 con 0x7f6ac0064e10 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac0114020 con 0x7f6ac0064e10 2023-12-19T18:46:12.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ab67fc640 1 -- 172.21.15.59:0/3325918026 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6abc0094b0 con 0x7f6ac0064e10 2023-12-19T18:46:12.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.426+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac0114590 con 0x7f6ac0064e10 2023-12-19T18:46:12.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.427+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac0106ba0 con 0x7f6ac0064e10 2023-12-19T18:46:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.427+0000 7f6ab67fc640 1 -- 172.21.15.59:0/3325918026 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6abc009720 con 0x7f6ac0064e10 2023-12-19T18:46:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.428+0000 7f6ab67fc640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6aac068ee0 0x7f6aac06b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.428+0000 7f6ab67fc640 1 -- 172.21.15.59:0/3325918026 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6abc086b80 con 0x7f6ac0064e10 2023-12-19T18:46:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.428+0000 7f6ac54ef640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6aac068ee0 0x7f6aac06b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:12.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.428+0000 7f6ac54ef640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6aac068ee0 0x7f6aac06b390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00047f0 tx=0x7f6ab0009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:12.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.432+0000 7f6ab67fc640 1 -- 172.21.15.59:0/3325918026 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6abc052f80 con 0x7f6ac0064e10 2023-12-19T18:46:12.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.605+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f6ac0114780 con 0x7f6ac0064e10 2023-12-19T18:46:12.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.605+0000 7f6ab67fc640 1 -- 172.21.15.59:0/3325918026 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6abc0565a0 con 0x7f6ac0064e10 2023-12-19T18:46:12.606 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2023-12-19T18:46:12.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6aac068ee0 msgr2=0x7f6aac06b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6aac068ee0 0x7f6aac06b390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00047f0 tx=0x7f6ab0009da0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac0064e10 msgr2=0x7f6ac010e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac0064e10 0x7f6ac010e900 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00eb70 tx=0x7f6abc00b040 comp rx=0 tx=0).stop 2023-12-19T18:46:12.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 shutdown_connections 2023-12-19T18:46:12.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6aac068ee0 0x7f6aac06b390 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6ac01014b0 0x7f6ac010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 --2- 172.21.15.59:0/3325918026 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ac0064e10 0x7f6ac010e900 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 >> 172.21.15.59:0/3325918026 conn(0x7f6ac00f6580 msgr2=0x7f6ac00f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:12.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 shutdown_connections 2023-12-19T18:46:12.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.607+0000 7f6ac7f54640 1 -- 172.21.15.59:0/3325918026 wait complete. 2023-12-19T18:46:12.623 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2023-12-19T18:46:12.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.733+0000 7f706dc07640 1 -- 172.21.15.59:0/3203263798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70680ff110 msgr2=0x7f7068101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.733+0000 7f706dc07640 1 --2- 172.21.15.59:0/3203263798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70680ff110 0x7f7068101550 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f705c0048d0 tx=0x7f705c009da0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.733+0000 7f706dc07640 1 -- 172.21.15.59:0/3203263798 shutdown_connections 2023-12-19T18:46:12.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.733+0000 7f706dc07640 1 --2- 172.21.15.59:0/3203263798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f70680ff110 0x7f7068101550 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.733+0000 7f706dc07640 1 --2- 172.21.15.59:0/3203263798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7068065ab0 0x7f70680feb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.733+0000 7f706dc07640 1 -- 172.21.15.59:0/3203263798 >> 172.21.15.59:0/3203263798 conn(0x7f70680f65b0 msgr2=0x7f70680f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.733+0000 7f706dc07640 1 -- 172.21.15.59:0/3203263798 shutdown_connections 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.733+0000 7f706dc07640 1 -- 172.21.15.59:0/3203263798 wait complete. 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f706dc07640 1 Processor -- start 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f706dc07640 1 -- start start 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f706dc07640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7068065ab0 0x7f706810c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f706dc07640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680ff110 0x7f706810cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f706dc07640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706810d2f0 con 0x7f7068065ab0 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f706dc07640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7068111d40 con 0x7f70680ff110 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f70677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7068065ab0 0x7f706810c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f70677fe640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7068065ab0 0x7f706810c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35946/0 (socket says 172.21.15.59:35946) 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f70677fe640 1 -- 172.21.15.59:0/4073046061 learned_addr learned my addr 172.21.15.59:0/4073046061 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f7066ffd640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680ff110 0x7f706810cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:12.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f70677fe640 1 -- 172.21.15.59:0/4073046061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680ff110 msgr2=0x7f706810cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.734+0000 7f70677fe640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680ff110 0x7f706810cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.735+0000 7f70677fe640 1 -- 172.21.15.59:0/4073046061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c005040 con 0x7f7068065ab0 2023-12-19T18:46:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.735+0000 7f7066ffd640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680ff110 0x7f706810cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T18:46:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.735+0000 7f70677fe640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7068065ab0 0x7f706810c760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7058002a50 tx=0x7f7058004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.735+0000 7f704ffff640 1 -- 172.21.15.59:0/4073046061 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7058007c30 con 0x7f7068065ab0 2023-12-19T18:46:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.735+0000 7f704ffff640 1 -- 172.21.15.59:0/4073046061 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f705800f040 con 0x7f7068065ab0 2023-12-19T18:46:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.735+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7068111ee0 con 0x7f7068065ab0 2023-12-19T18:46:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.735+0000 7f704ffff640 1 -- 172.21.15.59:0/4073046061 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f705800daf0 con 0x7f7068065ab0 2023-12-19T18:46:12.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.735+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70681123f0 con 0x7f7068065ab0 2023-12-19T18:46:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.736+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70681049b0 con 0x7f7068065ab0 2023-12-19T18:46:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.737+0000 7f704ffff640 1 -- 172.21.15.59:0/4073046061 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f705800dc90 con 0x7f7068065ab0 2023-12-19T18:46:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.737+0000 7f704ffff640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7050068ed0 0x7f705006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.737+0000 7f7066ffd640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7050068ed0 0x7f705006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.737+0000 7f704ffff640 1 -- 172.21.15.59:0/4073046061 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7058086730 con 0x7f7068065ab0 2023-12-19T18:46:12.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.738+0000 7f7066ffd640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7050068ed0 0x7f705006b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f705c000c00 tx=0x7f705c009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:12.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.742+0000 7f704ffff640 1 -- 172.21.15.59:0/4073046061 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7058052bf0 con 0x7f7068065ab0 2023-12-19T18:46:12.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.924+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f7068112890 con 0x7f7068065ab0 2023-12-19T18:46:12.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.924+0000 7f704ffff640 1 -- 172.21.15.59:0/4073046061 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f7058056210 con 0x7f7068065ab0 2023-12-19T18:46:12.925 INFO:teuthology.orchestra.run.smithi059.stdout:pg_num: 32 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7050068ed0 msgr2=0x7f705006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7050068ed0 0x7f705006b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f705c000c00 tx=0x7f705c009da0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7068065ab0 msgr2=0x7f706810c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7068065ab0 0x7f706810c760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7058002a50 tx=0x7f7058004600 comp rx=0 tx=0).stop 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 shutdown_connections 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7050068ed0 0x7f705006b380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f70680ff110 0x7f706810cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 --2- 172.21.15.59:0/4073046061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7068065ab0 0x7f706810c760 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 >> 172.21.15.59:0/4073046061 conn(0x7f70680f65b0 msgr2=0x7f70680f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:12.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 shutdown_connections 2023-12-19T18:46:12.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:12.926+0000 7f706dc07640 1 -- 172.21.15.59:0/4073046061 wait complete. 2023-12-19T18:46:12.938 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T18:46:13.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.042+0000 7fac9b0ae640 1 -- 172.21.15.59:0/3791440164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 msgr2=0x7fac940fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:13.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.042+0000 7fac9b0ae640 1 --2- 172.21.15.59:0/3791440164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 0x7fac940fb280 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009c50 tx=0x7fac8c004d10 comp rx=0 tx=0).stop 2023-12-19T18:46:13.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.042+0000 7fac9b0ae640 1 -- 172.21.15.59:0/3791440164 shutdown_connections 2023-12-19T18:46:13.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.042+0000 7fac9b0ae640 1 --2- 172.21.15.59:0/3791440164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac940fc090 0x7fac940fc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:13.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.042+0000 7fac9b0ae640 1 --2- 172.21.15.59:0/3791440164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 0x7fac940fb280 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:13.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.042+0000 7fac9b0ae640 1 -- 172.21.15.59:0/3791440164 >> 172.21.15.59:0/3791440164 conn(0x7fac940f65c0 msgr2=0x7fac940f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:13.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.042+0000 7fac9b0ae640 1 -- 172.21.15.59:0/3791440164 shutdown_connections 2023-12-19T18:46:13.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.042+0000 7fac9b0ae640 1 -- 172.21.15.59:0/3791440164 wait complete. 2023-12-19T18:46:13.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac9b0ae640 1 Processor -- start 2023-12-19T18:46:13.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac9b0ae640 1 -- start start 2023-12-19T18:46:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac9b0ae640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 0x7fac94065e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac9b0ae640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac940fc090 0x7fac941139c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac9b0ae640 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9410d3a0 con 0x7fac940fae70 2023-12-19T18:46:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac9b0ae640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94113f00 con 0x7fac940fc090 2023-12-19T18:46:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac98e4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 0x7fac94065e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac98e4a640 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 0x7fac94065e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35974/0 (socket says 172.21.15.59:35974) 2023-12-19T18:46:13.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac98e4a640 1 -- 172.21.15.59:0/839131558 learned_addr learned my addr 172.21.15.59:0/839131558 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.043+0000 7fac93fff640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac940fc090 0x7fac941139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.044+0000 7fac98e4a640 1 -- 172.21.15.59:0/839131558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac940fc090 msgr2=0x7fac941139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.044+0000 7fac98e4a640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac940fc090 0x7fac941139c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.044+0000 7fac98e4a640 1 -- 172.21.15.59:0/839131558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c0098b0 con 0x7fac940fae70 2023-12-19T18:46:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.044+0000 7fac98e4a640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 0x7fac94065e50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fac8c003400 tx=0x7fac8c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:13.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.044+0000 7fac917fa640 1 -- 172.21.15.59:0/839131558 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac8c03a070 con 0x7fac940fae70 2023-12-19T18:46:13.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.044+0000 7fac917fa640 1 -- 172.21.15.59:0/839131558 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac8c0094c0 con 0x7fac940fae70 2023-12-19T18:46:13.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.044+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac941140a0 con 0x7fac940fae70 2023-12-19T18:46:13.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.044+0000 7fac917fa640 1 -- 172.21.15.59:0/839131558 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac8c03e650 con 0x7fac940fae70 2023-12-19T18:46:13.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.044+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac94114610 con 0x7fac940fae70 2023-12-19T18:46:13.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.045+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac94105800 con 0x7fac940fae70 2023-12-19T18:46:13.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.045+0000 7fac917fa640 1 -- 172.21.15.59:0/839131558 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fac8c032070 con 0x7fac940fae70 2023-12-19T18:46:13.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.046+0000 7fac917fa640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fac80068ed0 0x7fac8006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:13.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.046+0000 7fac917fa640 1 -- 172.21.15.59:0/839131558 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fac8c0a8b30 con 0x7fac940fae70 2023-12-19T18:46:13.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.046+0000 7fac93fff640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fac80068ed0 0x7fac8006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:13.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.046+0000 7fac93fff640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fac80068ed0 0x7fac8006b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fac94065c00 tx=0x7fac8400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:13.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.050+0000 7fac917fa640 1 -- 172.21.15.59:0/839131558 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fac8c074f30 con 0x7fac940fae70 2023-12-19T18:46:13.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:12 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3796773394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T18:46:13.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:12 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3328049496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-19T18:46:13.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:12 smithi059 ceph-mon[80661]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2023-12-19T18:46:13.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:12 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3325918026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2023-12-19T18:46:13.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:12 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/4073046061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2023-12-19T18:46:13.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:12 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3796773394' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T18:46:13.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:12 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3328049496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-19T18:46:13.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:12 smithi089 ceph-mon[88459]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2023-12-19T18:46:13.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:12 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3325918026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2023-12-19T18:46:13.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:12 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/4073046061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2023-12-19T18:46:13.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.294+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac94103850 con 0x7fac940fae70 2023-12-19T18:46:13.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.294+0000 7fac917fa640 1 -- 172.21.15.59:0/839131558 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4685 (secure 0 0 0) 0x7fac8c078550 con 0x7fac940fae70 2023-12-19T18:46:13.295 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:13.295 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14486,"name":"cephfs.smithi059.abqklt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.59:6827/1872103080","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6826","nonce":1872103080},{"type":"v1","addr":"172.21.15.59:6827","nonce":1872103080}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24291,"name":"cephfs.smithi089.dpgsbt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.89:6825/1622105230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":1622105230},{"type":"v1","addr":"172.21.15.89:6825","nonce":1622105230}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24301,"name":"cephfs.smithi089.dfrkla","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.89:6827/3691364030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":3691364030},{"type":"v1","addr":"172.21.15.89:6827","nonce":3691364030}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T18:45:45.631056+0000","modified":"2023-12-19T18:46:03.615908+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14474},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14474":{"gid":14474,"name":"cephfs.smithi059.rtqzmc","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.59:6829/222794355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6828","nonce":222794355},{"type":"v1","addr":"172.21.15.59:6829","nonce":222794355}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-12-19T18:46:13.295 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 15 2023-12-19T18:46:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fac80068ed0 msgr2=0x7fac8006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fac80068ed0 0x7fac8006b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fac94065c00 tx=0x7fac8400b040 comp rx=0 tx=0).stop 2023-12-19T18:46:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 msgr2=0x7fac94065e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 0x7fac94065e50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fac8c003400 tx=0x7fac8c004d10 comp rx=0 tx=0).stop 2023-12-19T18:46:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 shutdown_connections 2023-12-19T18:46:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fac80068ed0 0x7fac8006b380 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac940fc090 0x7fac941139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:13.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 --2- 172.21.15.59:0/839131558 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fac940fae70 0x7fac94065e50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:13.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 >> 172.21.15.59:0/839131558 conn(0x7fac940f65c0 msgr2=0x7fac940f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:13.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 shutdown_connections 2023-12-19T18:46:13.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:13.296+0000 7fac9b0ae640 1 -- 172.21.15.59:0/839131558 wait complete. 2023-12-19T18:46:13.308 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 18} 2023-12-19T18:46:13.308 INFO:teuthology.run_tasks:Running task kclient... 2023-12-19T18:46:13.335 INFO:tasks.kclient:Mounting kernel clients... 2023-12-19T18:46:13.335 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2023-12-19T18:46:13.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:13.335 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2023-12-19T18:46:13.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:13.352 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link delete ceph-brx 2023-12-19T18:46:13.427 INFO:teuthology.orchestra.run.smithi059.stderr:Cannot find device "ceph-brx" 2023-12-19T18:46:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:46:13.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:13.429 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-12-19T18:46:13.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:13.449 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2023-12-19T18:46:13.526 INFO:teuthology.orchestra.run.smithi089.stderr:Cannot find device "ceph-brx" 2023-12-19T18:46:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:46:13.528 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-12-19T18:46:13.528 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-19T18:46:13.528 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T18:46:13.528 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph fs ls 2023-12-19T18:46:13.939 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:14.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:13 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/839131558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-19T18:46:14.257 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:13 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/839131558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-19T18:46:15.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:14 smithi089 ceph-mon[88459]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2023-12-19T18:46:15.259 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:14 smithi059 ceph-mon[80661]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2023-12-19T18:46:15.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 -- 172.21.15.59:0/3063820215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b64101b00 msgr2=0x7f1b64101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 --2- 172.21.15.59:0/3063820215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b64101b00 0x7f1b64101ed0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1b54004940 tx=0x7f1b54009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 -- 172.21.15.59:0/3063820215 shutdown_connections 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 --2- 172.21.15.59:0/3063820215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b640fbb40 0x7f1b640fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 --2- 172.21.15.59:0/3063820215 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b64101b00 0x7f1b64101ed0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 -- 172.21.15.59:0/3063820215 >> 172.21.15.59:0/3063820215 conn(0x7f1b640f76c0 msgr2=0x7f1b640f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 -- 172.21.15.59:0/3063820215 shutdown_connections 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 -- 172.21.15.59:0/3063820215 wait complete. 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 Processor -- start 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.484+0000 7f1b68eeb700 1 -- start start 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b68eeb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b640fbb40 0x7f1b6410e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b68eeb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b64101b00 0x7f1b6410ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b68eeb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6410f260 con 0x7f1b640fbb40 2023-12-19T18:46:15.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b68eeb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6410f3d0 con 0x7f1b64101b00 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b6259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b640fbb40 0x7f1b6410e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b6259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b640fbb40 0x7f1b6410e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35988/0 (socket says 172.21.15.59:35988) 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b6259c700 1 -- 172.21.15.59:0/887971009 learned_addr learned my addr 172.21.15.59:0/887971009 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b61d9b700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b64101b00 0x7f1b6410ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b6259c700 1 -- 172.21.15.59:0/887971009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b64101b00 msgr2=0x7f1b6410ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b6259c700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b64101b00 0x7f1b6410ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b6259c700 1 -- 172.21.15.59:0/887971009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b54005040 con 0x7f1b640fbb40 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b61d9b700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b64101b00 0x7f1b6410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b6259c700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b640fbb40 0x7f1b6410e700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009390 tx=0x7f1b54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b52ffd700 1 -- 172.21.15.59:0/887971009 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b5400e040 con 0x7f1b640fbb40 2023-12-19T18:46:15.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b52ffd700 1 -- 172.21.15.59:0/887971009 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b5400bc50 con 0x7f1b640fbb40 2023-12-19T18:46:15.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b641176c0 con 0x7f1b640fbb40 2023-12-19T18:46:15.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b52ffd700 1 -- 172.21.15.59:0/887971009 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b5401c630 con 0x7f1b640fbb40 2023-12-19T18:46:15.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.485+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b64117bd0 con 0x7f1b640fbb40 2023-12-19T18:46:15.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.486+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b64103fb0 con 0x7f1b640fbb40 2023-12-19T18:46:15.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.487+0000 7f1b52ffd700 1 -- 172.21.15.59:0/887971009 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1b54013420 con 0x7f1b640fbb40 2023-12-19T18:46:15.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.487+0000 7f1b52ffd700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1b4c068f70 0x7f1b4c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:15.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.488+0000 7f1b61d9b700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1b4c068f70 0x7f1b4c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:15.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.488+0000 7f1b52ffd700 1 -- 172.21.15.59:0/887971009 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1b54088120 con 0x7f1b640fbb40 2023-12-19T18:46:15.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.488+0000 7f1b61d9b700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1b4c068f70 0x7f1b4c06b420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1b58006fd0 tx=0x7f1b58008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:15.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.492+0000 7f1b52ffd700 1 -- 172.21.15.59:0/887971009 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1b540545a0 con 0x7f1b640fbb40 2023-12-19T18:46:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.735+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1b64066420 con 0x7f1b640fbb40 2023-12-19T18:46:15.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.736+0000 7f1b52ffd700 1 -- 172.21.15.59:0/887971009 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f1b54057bc0 con 0x7f1b640fbb40 2023-12-19T18:46:15.737 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2023-12-19T18:46:15.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1b4c068f70 msgr2=0x7f1b4c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1b4c068f70 0x7f1b4c06b420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1b58006fd0 tx=0x7f1b58008040 comp rx=0 tx=0).stop 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b640fbb40 msgr2=0x7f1b6410e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b640fbb40 0x7f1b6410e700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009390 tx=0x7f1b54009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 shutdown_connections 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f1b4c068f70 0x7f1b4c06b420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b640fbb40 0x7f1b6410e700 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 --2- 172.21.15.59:0/887971009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b64101b00 0x7f1b6410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 >> 172.21.15.59:0/887971009 conn(0x7f1b640f76c0 msgr2=0x7f1b640f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 shutdown_connections 2023-12-19T18:46:15.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:15.738+0000 7f1b68eeb700 1 -- 172.21.15.59:0/887971009 wait complete. 2023-12-19T18:46:16.434 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T18:46:16.434 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T18:46:16.434 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi059.front.sepia.ceph.com 2023-12-19T18:46:16.434 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T18:46:16.434 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T18:46:16.434 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T18:46:16.434 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T18:46:16.434 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T18:46:16.434 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T18:46:16.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:16.435 DEBUG:teuthology.orchestra.run.smithi059:> ip addr 2023-12-19T18:46:16.453 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T18:46:16.453 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T18:46:16.453 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42181sec preferred_lft 42181sec 2023-12-19T18:46:16.454 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link noprefixroute 2023-12-19T18:46:16.455 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2023-12-19T18:46:16.455 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T18:46:16.455 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2023-12-19T18:46:16.455 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2023-12-19T18:46:16.455 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T18:46:16.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:16.455 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2023-12-19T18:46:16.456 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2023-12-19T18:46:16.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add name ceph-brx type bridge 2023-12-19T18:46:16.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr flush dev ceph-brx 2023-12-19T18:46:16.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set ceph-brx up 2023-12-19T18:46:16.456 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T18:46:16.456 DEBUG:teuthology.orchestra.run.smithi059:> ') 2023-12-19T18:46:16.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:16.633 DEBUG:teuthology.orchestra.run.smithi059:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T18:46:16.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:16 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:16.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:16 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:16.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:16 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:46:16.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:16 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:46:16.661 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:16 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:16.662 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:16 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/887971009' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-19T18:46:16.663 INFO:teuthology.orchestra.run.smithi059.stdout:1 2023-12-19T18:46:16.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:16.665 DEBUG:teuthology.orchestra.run.smithi059:> route 2023-12-19T18:46:16.734 INFO:teuthology.orchestra.run.smithi059.stdout:Kernel IP routing table 2023-12-19T18:46:16.734 INFO:teuthology.orchestra.run.smithi059.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T18:46:16.734 INFO:teuthology.orchestra.run.smithi059.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-19T18:46:16.734 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-19T18:46:16.734 INFO:teuthology.orchestra.run.smithi059.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T18:46:16.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:16.735 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2023-12-19T18:46:16.735 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2023-12-19T18:46:16.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T18:46:16.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T18:46:16.735 DEBUG:teuthology.orchestra.run.smithi059:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T18:46:16.735 DEBUG:teuthology.orchestra.run.smithi059:> ') 2023-12-19T18:46:17.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:16 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:17.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:16 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:17.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:16 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:46:17.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:16 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:46:17.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:16 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:17.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:16 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/887971009' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-19T18:46:17.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.019 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list 2023-12-19T18:46:17.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.036 DEBUG:teuthology.orchestra.run.smithi059:> ip netns list-id 2023-12-19T18:46:17.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.093 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2023-12-19T18:46:17.093 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2023-12-19T18:46:17.093 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T18:46:17.093 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-19T18:46:17.093 DEBUG:teuthology.orchestra.run.smithi059:> ') 2023-12-19T18:46:17.183 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-19T18:46:17.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.183 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2023-12-19T18:46:17.183 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2023-12-19T18:46:17.183 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-19T18:46:17.183 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T18:46:17.183 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-19T18:46:17.183 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-19T18:46:17.184 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-19T18:46:17.184 DEBUG:teuthology.orchestra.run.smithi059:> ') 2023-12-19T18:46:17.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.298 DEBUG:teuthology.orchestra.run.smithi059:> (cd / && exec bash -c ' 2023-12-19T18:46:17.299 DEBUG:teuthology.orchestra.run.smithi059:> set -e 2023-12-19T18:46:17.299 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set brx.0 up 2023-12-19T18:46:17.299 DEBUG:teuthology.orchestra.run.smithi059:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T18:46:17.299 DEBUG:teuthology.orchestra.run.smithi059:> ') 2023-12-19T18:46:17.348 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T18:46:17.348 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T18:46:17.363 INFO:teuthology.orchestra.run.smithi059.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T18:46:17.364 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T18:46:17.364 DEBUG:teuthology.orchestra.run.smithi059:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T18:46:17.419 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-12-19T18:46:17.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.419 DEBUG:teuthology.orchestra.run.smithi059:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-19T18:46:17.511 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-19T18:46:17.512 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-19T18:46:17.512 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2023-12-19T18:46:17.615 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-19T18:46:17.615 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-19T18:46:17.615 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2023-12-19T18:46:17.615 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.89:3300" 2023-12-19T18:46:17.615 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-19T18:46:17.620 INFO:teuthology.orchestra.run.smithi059.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-19T18:46:17.620 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-19T18:46:17.620 INFO:teuthology.orchestra.run.smithi059.stdout:invalid new device string format 2023-12-19T18:46:17.620 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.89:3300" 2023-12-19T18:46:17.620 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-19T18:46:17.620 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.89:6789" 2023-12-19T18:46:17.620 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.89:6789:/ 2023-12-19T18:46:17.620 INFO:teuthology.orchestra.run.smithi059.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=bf854274-9e9d-11ee-95a5-87774f69a715" will pass to kernel 2023-12-19T18:46:17.621 INFO:teuthology.orchestra.run.smithi059.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-12-19T18:46:17.621 INFO:teuthology.orchestra.run.smithi059.stdout: You just mounted a file system that supports labels which does not 2023-12-19T18:46:17.621 INFO:teuthology.orchestra.run.smithi059.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-12-19T18:46:17.621 INFO:teuthology.orchestra.run.smithi059.stdout: applications will generate AVC messages and not be allowed access to 2023-12-19T18:46:17.621 INFO:teuthology.orchestra.run.smithi059.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-12-19T18:46:17.622 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-12-19T18:46:17.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.623 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T18:46:17.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.655 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T18:46:17.715 INFO:teuthology.orchestra.run.smithi059.stdout:client14524 2023-12-19T18:46:17.716 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:46:17.716 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T18:46:17.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.773 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T18:46:17.834 INFO:teuthology.orchestra.run.smithi059.stdout:client14524 2023-12-19T18:46:17.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.835 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T18:46:17.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:17 smithi089 ceph-mon[88459]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 3 op/s 2023-12-19T18:46:17.895 INFO:teuthology.orchestra.run.smithi059.stdout:client14524 2023-12-19T18:46:17.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:17.896 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/bf854274-9e9d-11ee-95a5-87774f69a715.client14524/status') 2023-12-19T18:46:17.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:17 smithi059 ceph-mon[80661]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.4 KiB/s rd, 0 B/s wr, 3 op/s 2023-12-19T18:46:17.969 INFO:teuthology.orchestra.run.smithi059.stdout:instance: client.14524 (3)192.168.0.1:0/3702317007 2023-12-19T18:46:17.969 INFO:teuthology.orchestra.run.smithi059.stdout:blocklisted: false 2023-12-19T18:46:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records in 2023-12-19T18:46:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:0+1 records out 2023-12-19T18:46:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:70 bytes copied, 2.9192e-05 s, 2.4 MB/s 2023-12-19T18:46:17.971 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-12-19T18:46:17.971 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-19T18:46:17.971 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-19T18:46:17.971 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -- ceph fs ls 2023-12-19T18:46:18.281 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:19.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:19 smithi089 ceph-mon[88459]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.806+0000 7fd61665e700 1 -- 172.21.15.59:0/727446172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd610102c00 msgr2=0x7fd610102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.806+0000 7fd61665e700 1 --2- 172.21.15.59:0/727446172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd610102c00 0x7fd610102fd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd600004940 tx=0x7fd600009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.806+0000 7fd61665e700 1 -- 172.21.15.59:0/727446172 shutdown_connections 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.806+0000 7fd61665e700 1 --2- 172.21.15.59:0/727446172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6101035a0 0x7fd6100fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.806+0000 7fd61665e700 1 --2- 172.21.15.59:0/727446172 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd610102c00 0x7fd610102fd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.806+0000 7fd61665e700 1 -- 172.21.15.59:0/727446172 >> 172.21.15.59:0/727446172 conn(0x7fd6100f8790 msgr2=0x7fd6100faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.806+0000 7fd61665e700 1 -- 172.21.15.59:0/727446172 shutdown_connections 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.806+0000 7fd61665e700 1 -- 172.21.15.59:0/727446172 wait complete. 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.807+0000 7fd61665e700 1 Processor -- start 2023-12-19T18:46:19.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.807+0000 7fd61665e700 1 -- start start 2023-12-19T18:46:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.807+0000 7fd61665e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd610102c00 0x7fd610113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.807+0000 7fd61665e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6101035a0 0x7fd610113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.807+0000 7fd61665e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd610114200 con 0x7fd6101035a0 2023-12-19T18:46:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.807+0000 7fd61665e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61010d4f0 con 0x7fd610102c00 2023-12-19T18:46:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6101035a0 0x7fd610113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6101035a0 0x7fd610113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36010/0 (socket says 172.21.15.59:36010) 2023-12-19T18:46:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60f7fe700 1 -- 172.21.15.59:0/3569961613 learned_addr learned my addr 172.21.15.59:0/3569961613 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60ffff700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd610102c00 0x7fd610113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:19.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60f7fe700 1 -- 172.21.15.59:0/3569961613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd610102c00 msgr2=0x7fd610113630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:19.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60f7fe700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd610102c00 0x7fd610113630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:19.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60f7fe700 1 -- 172.21.15.59:0/3569961613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd600005040 con 0x7fd6101035a0 2023-12-19T18:46:19.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60f7fe700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6101035a0 0x7fd610113bb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd60400c800 tx=0x7fd604004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:19.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60cff9700 1 -- 172.21.15.59:0/3569961613 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6040106f0 con 0x7fd6101035a0 2023-12-19T18:46:19.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61010d780 con 0x7fd6101035a0 2023-12-19T18:46:19.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60cff9700 1 -- 172.21.15.59:0/3569961613 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd604010d00 con 0x7fd6101035a0 2023-12-19T18:46:19.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd60cff9700 1 -- 172.21.15.59:0/3569961613 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6040113a0 con 0x7fd6101035a0 2023-12-19T18:46:19.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.808+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61010dcc0 con 0x7fd6101035a0 2023-12-19T18:46:19.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.809+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6100fff10 con 0x7fd6101035a0 2023-12-19T18:46:19.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.810+0000 7fd60cff9700 1 -- 172.21.15.59:0/3569961613 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd604012400 con 0x7fd6101035a0 2023-12-19T18:46:19.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.810+0000 7fd60cff9700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd5f8068f60 0x7fd5f806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:19.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.810+0000 7fd60cff9700 1 -- 172.21.15.59:0/3569961613 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd604086d80 con 0x7fd6101035a0 2023-12-19T18:46:19.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.810+0000 7fd60ffff700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd5f8068f60 0x7fd5f806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:19.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.811+0000 7fd60ffff700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd5f8068f60 0x7fd5f806b410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd600004f90 tx=0x7fd600009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:19.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:19.815+0000 7fd60cff9700 1 -- 172.21.15.59:0/3569961613 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd604056c90 con 0x7fd6101035a0 2023-12-19T18:46:19.816 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:19 smithi059 ceph-mon[80661]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2023-12-19T18:46:20.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.060+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd61004bbb0 con 0x7fd6101035a0 2023-12-19T18:46:20.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.061+0000 7fd60cff9700 1 -- 172.21.15.59:0/3569961613 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7fd604056820 con 0x7fd6101035a0 2023-12-19T18:46:20.062 INFO:teuthology.orchestra.run.smithi059.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2023-12-19T18:46:20.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd5f8068f60 msgr2=0x7fd5f806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:20.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd5f8068f60 0x7fd5f806b410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd600004f90 tx=0x7fd600009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:20.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6101035a0 msgr2=0x7fd610113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:20.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6101035a0 0x7fd610113bb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd60400c800 tx=0x7fd604004470 comp rx=0 tx=0).stop 2023-12-19T18:46:20.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 shutdown_connections 2023-12-19T18:46:20.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd610102c00 0x7fd610113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:20.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd5f8068f60 0x7fd5f806b410 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:20.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 --2- 172.21.15.59:0/3569961613 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd6101035a0 0x7fd610113bb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:20.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 >> 172.21.15.59:0/3569961613 conn(0x7fd6100f8790 msgr2=0x7fd6100fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:20.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 shutdown_connections 2023-12-19T18:46:20.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:20.063+0000 7fd61665e700 1 -- 172.21.15.59:0/3569961613 wait complete. 2023-12-19T18:46:20.692 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:20 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3569961613' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-19T18:46:20.721 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T18:46:20.722 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T18:46:20.722 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2023-12-19T18:46:20.722 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-12-19T18:46:20.722 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-19T18:46:20.722 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T18:46:20.722 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T18:46:20.722 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T18:46:20.722 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-12-19T18:46:20.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:20.722 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2023-12-19T18:46:20.741 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T18:46:20.741 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T18:46:20.741 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T18:46:20.741 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-12-19T18:46:20.741 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2023-12-19T18:46:20.741 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-12-19T18:46:20.741 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T18:46:20.741 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2023-12-19T18:46:20.741 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42159sec preferred_lft 42159sec 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link 2023-12-19T18:46:20.742 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2023-12-19T18:46:20.742 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T18:46:20.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:20.743 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-12-19T18:46:20.743 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-12-19T18:46:20.743 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add name ceph-brx type bridge 2023-12-19T18:46:20.743 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr flush dev ceph-brx 2023-12-19T18:46:20.743 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set ceph-brx up 2023-12-19T18:46:20.743 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T18:46:20.743 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-12-19T18:46:20.879 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:20 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3569961613' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-19T18:46:20.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:20.923 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T18:46:20.950 INFO:teuthology.orchestra.run.smithi089.stdout:1 2023-12-19T18:46:20.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:20.951 DEBUG:teuthology.orchestra.run.smithi089:> route 2023-12-19T18:46:21.028 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2023-12-19T18:46:21.028 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T18:46:21.028 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T18:46:21.028 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-19T18:46:21.028 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T18:46:21.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:21.029 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-12-19T18:46:21.029 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-12-19T18:46:21.029 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T18:46:21.029 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T18:46:21.029 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T18:46:21.030 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-12-19T18:46:21.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:21.265 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2023-12-19T18:46:21.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:21.280 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2023-12-19T18:46:21.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:21.335 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-12-19T18:46:21.335 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-12-19T18:46:21.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-19T18:46:21.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-12-19T18:46:21.335 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-12-19T18:46:21.420 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-12-19T18:46:21.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:21.420 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-12-19T18:46:21.420 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-12-19T18:46:21.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-12-19T18:46:21.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T18:46:21.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-12-19T18:46:21.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-12-19T18:46:21.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-12-19T18:46:21.420 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-12-19T18:46:21.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:21.563 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec bash -c ' 2023-12-19T18:46:21.563 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2023-12-19T18:46:21.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2023-12-19T18:46:21.563 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T18:46:21.563 DEBUG:teuthology.orchestra.run.smithi089:> ') 2023-12-19T18:46:21.610 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T18:46:21.610 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-12-19T18:46:21.637 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-12-19T18:46:21.637 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T18:46:21.637 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-12-19T18:46:21.692 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-12-19T18:46:21.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:21.692 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-19T18:46:21.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:21 smithi089 ceph-mon[88459]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2023-12-19T18:46:21.796 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-19T18:46:21.797 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-19T18:46:21.797 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-12-19T18:46:21.850 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:21 smithi059 ceph-mon[80661]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s rd, 0 B/s wr, 0 op/s 2023-12-19T18:46:21.904 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-19T18:46:21.904 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-19T18:46:21.904 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-12-19T18:46:21.904 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.89:3300" 2023-12-19T18:46:21.904 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-19T18:46:21.913 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-19T18:46:21.913 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-19T18:46:21.913 INFO:teuthology.orchestra.run.smithi089.stdout:invalid new device string format 2023-12-19T18:46:21.913 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.59:3300,172.21.15.89:3300" 2023-12-19T18:46:21.913 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-19T18:46:21.913 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.59:6789,172.21.15.89:6789" 2023-12-19T18:46:21.914 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.59:6789,172.21.15.89:6789:/ 2023-12-19T18:46:21.914 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=bf854274-9e9d-11ee-95a5-87774f69a715" will pass to kernel 2023-12-19T18:46:21.914 INFO:teuthology.orchestra.run.smithi089.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-12-19T18:46:21.914 INFO:teuthology.orchestra.run.smithi089.stdout: You just mounted a file system that supports labels which does not 2023-12-19T18:46:21.914 INFO:teuthology.orchestra.run.smithi089.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-12-19T18:46:21.914 INFO:teuthology.orchestra.run.smithi089.stdout: applications will generate AVC messages and not be allowed access to 2023-12-19T18:46:21.914 INFO:teuthology.orchestra.run.smithi089.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-12-19T18:46:21.920 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-12-19T18:46:21.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:21.920 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-12-19T18:46:21.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:21.989 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T18:46:22.045 INFO:teuthology.orchestra.run.smithi089.stdout:client14530 2023-12-19T18:46:22.046 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:46:22.046 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T18:46:22.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:22.102 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T18:46:22.158 INFO:teuthology.orchestra.run.smithi089.stdout:client14530 2023-12-19T18:46:22.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:22.158 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T18:46:22.217 INFO:teuthology.orchestra.run.smithi089.stdout:client14530 2023-12-19T18:46:22.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T18:46:22.217 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/bf854274-9e9d-11ee-95a5-87774f69a715.client14530/status') 2023-12-19T18:46:22.285 INFO:teuthology.orchestra.run.smithi089.stdout:instance: client.14530 (3)172.21.15.89:0/2544840139 2023-12-19T18:46:22.285 INFO:teuthology.orchestra.run.smithi089.stdout:blocklisted: false 2023-12-19T18:46:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records in 2023-12-19T18:46:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records out 2023-12-19T18:46:22.286 INFO:teuthology.orchestra.run.smithi089.stderr:71 bytes copied, 2.3571e-05 s, 3.0 MB/s 2023-12-19T18:46:22.287 INFO:teuthology.run_tasks:Running task print... 2023-12-19T18:46:22.297 INFO:teuthology.task.print:**** done client 2023-12-19T18:46:22.297 INFO:teuthology.run_tasks:Running task parallel... 2023-12-19T18:46:22.320 INFO:teuthology.task.parallel:starting parallel... 2023-12-19T18:46:22.320 INFO:teuthology.task.parallel:In parallel, running task sequential... 2023-12-19T18:46:22.321 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2023-12-19T18:46:22.321 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:46:22.321 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2023-12-19T18:46:22.322 INFO:teuthology.task.parallel:In parallel, running task sequential... 2023-12-19T18:46:22.322 INFO:teuthology.task.sequential:In sequential, running task workunit... 2023-12-19T18:46:22.326 INFO:tasks.workunit:Pulling workunits from ref c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:46:22.326 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-19T18:46:22.326 INFO:tasks.workunit:timeout=3h 2023-12-19T18:46:22.326 INFO:tasks.workunit:cleanup=True 2023-12-19T18:46:22.326 DEBUG:teuthology.orchestra.run.smithi059:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-19T18:46:22.349 INFO:teuthology.orchestra.run.smithi059.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-12-19T18:46:22.349 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-12-19T18:46:22.350 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 7eh/126d Inode: 1 Links: 2 2023-12-19T18:46:22.350 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:46:22.350 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:unlabeled_t:s0 2023-12-19T18:46:22.350 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-19T18:46:22.350 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2023-12-19 18:45:49.658208852 +0000 2023-12-19T18:46:22.350 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2023-12-19 18:46:17.652558735 +0000 2023-12-19T18:46:22.350 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: 2023-12-19 18:45:49.658208852 +0000 2023-12-19T18:46:22.350 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-12-19T18:46:22.350 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-12-19T18:46:22.426 DEBUG:teuthology.orchestra.run.smithi089:> stat -- /home/ubuntu/cephtest/mnt.1 2023-12-19T18:46:22.442 INFO:teuthology.orchestra.run.smithi089.stdout: File: /home/ubuntu/cephtest/mnt.1 2023-12-19T18:46:22.442 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2023-12-19T18:46:22.442 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 73h/115d Inode: 1 Links: 3 2023-12-19T18:46:22.442 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T18:46:22.443 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:unlabeled_t:s0 2023-12-19T18:46:22.443 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-19T18:46:22.443 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2023-12-19 18:46:22.420499095 +0000 2023-12-19T18:46:22.443 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2023-12-19 18:46:22.420499095 +0000 2023-12-19T18:46:22.443 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2023-12-19 18:45:49.658208852 +0000 2023-12-19T18:46:22.443 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2023-12-19T18:46:22.443 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2023-12-19T18:46:22.516 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:46:22.516 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:46:22.533 INFO:tasks.workunit.client.0.smithi059.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-19T18:46:22.571 INFO:tasks.workunit.client.1.smithi089.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2023-12-19T18:46:22.625 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:23.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:22 smithi059 ceph-mon[80661]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2023-12-19T18:46:23.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:22 smithi089 ceph-mon[88459]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2023-12-19T18:46:24.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.125+0000 7ff7ae2fe700 1 -- 172.21.15.59:0/2218731118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80fbaf0 msgr2=0x7ff7a80fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:24.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.125+0000 7ff7ae2fe700 1 --2- 172.21.15.59:0/2218731118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80fbaf0 0x7ff7a80fbf00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff798004940 tx=0x7ff798009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.125+0000 7ff7ae2fe700 1 -- 172.21.15.59:0/2218731118 shutdown_connections 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.125+0000 7ff7ae2fe700 1 --2- 172.21.15.59:0/2218731118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7a80fcd00 0x7ff7a80fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.125+0000 7ff7ae2fe700 1 --2- 172.21.15.59:0/2218731118 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80fbaf0 0x7ff7a80fbf00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.125+0000 7ff7ae2fe700 1 -- 172.21.15.59:0/2218731118 >> 172.21.15.59:0/2218731118 conn(0x7ff7a80f7190 msgr2=0x7ff7a80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.125+0000 7ff7ae2fe700 1 -- 172.21.15.59:0/2218731118 shutdown_connections 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.125+0000 7ff7ae2fe700 1 -- 172.21.15.59:0/2218731118 wait complete. 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7ae2fe700 1 Processor -- start 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7ae2fe700 1 -- start start 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7ae2fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7a80fbaf0 0x7ff7a810c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7ae2fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80fcd00 0x7ff7a810c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7ae2fe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a810cf30 con 0x7ff7a80fcd00 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7ae2fe700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a810d7d0 con 0x7ff7a80fbaf0 2023-12-19T18:46:24.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80fcd00 0x7ff7a810c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:24.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80fcd00 0x7ff7a810c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45584/0 (socket says 172.21.15.59:45584) 2023-12-19T18:46:24.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7a77fe700 1 -- 172.21.15.59:0/1507666188 learned_addr learned my addr 172.21.15.59:0/1507666188 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:24.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.126+0000 7ff7a7fff700 1 --2- 172.21.15.59:0/1507666188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7a80fbaf0 0x7ff7a810c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:24.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.127+0000 7ff7a77fe700 1 -- 172.21.15.59:0/1507666188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7a80fbaf0 msgr2=0x7ff7a810c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:24.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.127+0000 7ff7a77fe700 1 --2- 172.21.15.59:0/1507666188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7a80fbaf0 0x7ff7a810c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:24.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.127+0000 7ff7a77fe700 1 -- 172.21.15.59:0/1507666188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff798005040 con 0x7ff7a80fcd00 2023-12-19T18:46:24.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.127+0000 7ff7a77fe700 1 --2- 172.21.15.59:0/1507666188 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7a80fcd00 0x7ff7a810c930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff79c002a80 tx=0x7ff79c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.127+0000 7ff7a4ff9700 1 -- 172.21.15.59:0/1507666188 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff79c007cc0 con 0x7ff7a80fcd00 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.127+0000 7ff7a4ff9700 1 -- 172.21.15.59:0/1507666188 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff79c00f040 con 0x7ff7a80fcd00 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.127+0000 7ff7ae2fe700 1 -- 172.21.15.59:0/1507666188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a810dab0 con 0x7ff7a80fcd00 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.127+0000 7ff7a4ff9700 1 -- 172.21.15.59:0/1507666188 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff79c00da60 con 0x7ff7a80fcd00 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.127+0000 7ff7ae2fe700 1 -- 172.21.15.59:0/1507666188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a810dff0 con 0x7ff7a80fcd00 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.128+0000 7ff7ae2fe700 1 -- 172.21.15.59:0/1507666188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a804bb80 con 0x7ff7a80fcd00 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.129+0000 7ff7a4ff9700 1 -- 172.21.15.59:0/1507666188 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff79c00dbc0 con 0x7ff7a80fcd00 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.129+0000 7ff7a4ff9700 1 --2- 172.21.15.59:0/1507666188 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff790068f60 0x7ff79006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.129+0000 7ff7a4ff9700 1 -- 172.21.15.59:0/1507666188 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff79c086b70 con 0x7ff7a80fcd00 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.129+0000 7ff7a7fff700 1 --2- 172.21.15.59:0/1507666188 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff790068f60 0x7ff79006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:24.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.129+0000 7ff7a7fff700 1 --2- 172.21.15.59:0/1507666188 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff790068f60 0x7ff79006b410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff798004890 tx=0x7ff798009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:24.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.134+0000 7ff7a4ff9700 1 -- 172.21.15.59:0/1507666188 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff79c056a00 con 0x7ff7a80fcd00 2023-12-19T18:46:24.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:46:24.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:46:24.332 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.331+0000 7ff7ae2fe700 1 -- 172.21.15.59:0/1507666188 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7ff7a8104490 con 0x7ff7a80fcd00 2023-12-19T18:46:24.332 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:24.331+0000 7ff7a4ff9700 1 -- 172.21.15.59:0/1507666188 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7ff79c056590 con 0x7ff7a80fcd00 2023-12-19T18:46:24.334 INFO:teuthology.orchestra.run.smithi059.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2023-12-19T18:46:24.970 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2023-12-19T18:46:24.970 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:46:24.971 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2023-12-19T18:46:25.043 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:24 smithi059 ceph-mon[80661]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2023-12-19T18:46:25.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:24 smithi089 ceph-mon[88459]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2023-12-19T18:46:25.286 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:26.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.983+0000 7fce4c8bd700 1 -- 172.21.15.59:0/286951713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 msgr2=0x7fce480ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:26.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.983+0000 7fce4c8bd700 1 --2- 172.21.15.59:0/286951713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 0x7fce480ffb10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fce38004940 tx=0x7fce38009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:26.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.983+0000 7fce4c8bd700 1 -- 172.21.15.59:0/286951713 shutdown_connections 2023-12-19T18:46:26.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.983+0000 7fce4c8bd700 1 --2- 172.21.15.59:0/286951713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce48100050 0x7fce48102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.983+0000 7fce4c8bd700 1 --2- 172.21.15.59:0/286951713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 0x7fce480ffb10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.983+0000 7fce4c8bd700 1 -- 172.21.15.59:0/286951713 >> 172.21.15.59:0/286951713 conn(0x7fce480f71d0 msgr2=0x7fce480f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.983+0000 7fce4c8bd700 1 -- 172.21.15.59:0/286951713 shutdown_connections 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.983+0000 7fce4c8bd700 1 -- 172.21.15.59:0/286951713 wait complete. 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce4c8bd700 1 Processor -- start 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce4c8bd700 1 -- start start 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce4c8bd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 0x7fce480fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce4c8bd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce48100050 0x7fce480fb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce4c8bd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480fbc40 con 0x7fce480fd730 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce4c8bd700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480fbdb0 con 0x7fce48100050 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce4659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 0x7fce480fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce4659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 0x7fce480fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45606/0 (socket says 172.21.15.59:45606) 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce4659c700 1 -- 172.21.15.59:0/3769550652 learned_addr learned my addr 172.21.15.59:0/3769550652 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:26.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.984+0000 7fce45d9b700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce48100050 0x7fce480fb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce4659c700 1 -- 172.21.15.59:0/3769550652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce48100050 msgr2=0x7fce480fb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce4659c700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce48100050 0x7fce480fb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce4659c700 1 -- 172.21.15.59:0/3769550652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c005040 con 0x7fce480fd730 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce45d9b700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce48100050 0x7fce480fb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce4659c700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 0x7fce480fd1c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fce38009fd0 tx=0x7fce38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce36ffd700 1 -- 172.21.15.59:0/3769550652 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce38021070 con 0x7fce480fd730 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce36ffd700 1 -- 172.21.15.59:0/3769550652 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce3801c400 con 0x7fce480fd730 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce38005040 con 0x7fce480fd730 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce36ffd700 1 -- 172.21.15.59:0/3769550652 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce3800e070 con 0x7fce480fd730 2023-12-19T18:46:26.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.985+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce480fc380 con 0x7fce480fd730 2023-12-19T18:46:26.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.986+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4804bb80 con 0x7fce480fd730 2023-12-19T18:46:26.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.987+0000 7fce36ffd700 1 -- 172.21.15.59:0/3769550652 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fce3801ca00 con 0x7fce480fd730 2023-12-19T18:46:26.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.987+0000 7fce36ffd700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fce30068f60 0x7fce3006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:26.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.987+0000 7fce36ffd700 1 -- 172.21.15.59:0/3769550652 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fce38087f00 con 0x7fce480fd730 2023-12-19T18:46:26.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.987+0000 7fce45d9b700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fce30068f60 0x7fce3006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:26.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.988+0000 7fce45d9b700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fce30068f60 0x7fce3006b410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009fd0 tx=0x7fce3c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:26.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:26.992+0000 7fce36ffd700 1 -- 172.21.15.59:0/3769550652 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fce38057eb0 con 0x7fce480fd730 2023-12-19T18:46:27.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.194+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fce48105580 con 0x7fce480fd730 2023-12-19T18:46:27.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.195+0000 7fce36ffd700 1 -- 172.21.15.59:0/3769550652 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fce38057a40 con 0x7fce480fd730 2023-12-19T18:46:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fce30068f60 msgr2=0x7fce3006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fce30068f60 0x7fce3006b410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009fd0 tx=0x7fce3c00b040 comp rx=0 tx=0).stop 2023-12-19T18:46:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 msgr2=0x7fce480fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 0x7fce480fd1c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fce38009fd0 tx=0x7fce38009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 shutdown_connections 2023-12-19T18:46:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fce30068f60 0x7fce3006b410 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fce480fd730 0x7fce480fd1c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:27.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 --2- 172.21.15.59:0/3769550652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce48100050 0x7fce480fb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 >> 172.21.15.59:0/3769550652 conn(0x7fce480f71d0 msgr2=0x7fce480f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 shutdown_connections 2023-12-19T18:46:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:27.197+0000 7fce4c8bd700 1 -- 172.21.15.59:0/3769550652 wait complete. 2023-12-19T18:46:27.886 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:27 smithi089 ceph-mon[88459]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2023-12-19T18:46:27.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:27 smithi059 ceph-mon[80661]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2023-12-19T18:46:28.065 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2023-12-19T18:46:28.388 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:29.791 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:29 smithi059 ceph-mon[80661]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2023-12-19T18:46:30.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:29 smithi089 ceph-mon[88459]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.034+0000 7f5c963b0700 1 -- 172.21.15.59:0/3997009527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 msgr2=0x7f5c90102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.034+0000 7f5c963b0700 1 --2- 172.21.15.59:0/3997009527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 0x7f5c90102430 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5c84004940 tx=0x7f5c84009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.034+0000 7f5c963b0700 1 -- 172.21.15.59:0/3997009527 shutdown_connections 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.034+0000 7f5c963b0700 1 --2- 172.21.15.59:0/3997009527 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 0x7f5c90102430 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.034+0000 7f5c963b0700 1 --2- 172.21.15.59:0/3997009527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c900fd730 0x7f5c900ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.034+0000 7f5c963b0700 1 -- 172.21.15.59:0/3997009527 >> 172.21.15.59:0/3997009527 conn(0x7f5c900f7190 msgr2=0x7f5c900f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.034+0000 7f5c963b0700 1 -- 172.21.15.59:0/3997009527 shutdown_connections 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.034+0000 7f5c963b0700 1 -- 172.21.15.59:0/3997009527 wait complete. 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.034+0000 7f5c963b0700 1 Processor -- start 2023-12-19T18:46:30.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c963b0700 1 -- start start 2023-12-19T18:46:30.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c963b0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c900fd730 0x7f5c9010f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:30.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c963b0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 0x7f5c9010fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:30.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c963b0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c90110270 con 0x7f5c90100050 2023-12-19T18:46:30.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c963b0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c90110b10 con 0x7f5c900fd730 2023-12-19T18:46:30.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c8f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 0x7f5c9010fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:30.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c8ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c900fd730 0x7f5c9010f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:30.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c8ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c900fd730 0x7f5c9010f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.59:35540/0 (socket says 172.21.15.59:35540) 2023-12-19T18:46:30.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c8f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 0x7f5c9010fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45630/0 (socket says 172.21.15.59:45630) 2023-12-19T18:46:30.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c8ffff700 1 -- 172.21.15.59:0/855103284 learned_addr learned my addr 172.21.15.59:0/855103284 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:30.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c8f7fe700 1 -- 172.21.15.59:0/855103284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c900fd730 msgr2=0x7f5c9010f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:30.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c8f7fe700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c900fd730 0x7f5c9010f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:30.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.035+0000 7f5c8f7fe700 1 -- 172.21.15.59:0/855103284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c84005040 con 0x7f5c90100050 2023-12-19T18:46:30.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.036+0000 7f5c8f7fe700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 0x7f5c9010fc70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009fd0 tx=0x7f5c84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:30.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.036+0000 7f5c8cff9700 1 -- 172.21.15.59:0/855103284 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8400e040 con 0x7f5c90100050 2023-12-19T18:46:30.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.036+0000 7f5c8cff9700 1 -- 172.21.15.59:0/855103284 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c8400bc50 con 0x7f5c90100050 2023-12-19T18:46:30.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.036+0000 7f5c8cff9700 1 -- 172.21.15.59:0/855103284 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8401c650 con 0x7f5c90100050 2023-12-19T18:46:30.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.036+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c90110d90 con 0x7f5c90100050 2023-12-19T18:46:30.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.036+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c90111280 con 0x7f5c90100050 2023-12-19T18:46:30.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.037+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c900f8d30 con 0x7f5c90100050 2023-12-19T18:46:30.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.038+0000 7f5c8cff9700 1 -- 172.21.15.59:0/855103284 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f5c84013420 con 0x7f5c90100050 2023-12-19T18:46:30.040 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.038+0000 7f5c8cff9700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5c78068f60 0x7f5c7806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:30.040 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.038+0000 7f5c8cff9700 1 -- 172.21.15.59:0/855103284 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5c840880b0 con 0x7f5c90100050 2023-12-19T18:46:30.040 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.038+0000 7f5c8ffff700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5c78068f60 0x7f5c7806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:30.040 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.039+0000 7f5c8ffff700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5c78068f60 0x7f5c7806b410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5c800041d0 tx=0x7f5c8000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:30.043 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.043+0000 7f5c8cff9700 1 -- 172.21.15.59:0/855103284 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5c84057fc0 con 0x7f5c90100050 2023-12-19T18:46:30.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.277+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f5c9004bb80 con 0x7f5c90100050 2023-12-19T18:46:30.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.277+0000 7f5c8cff9700 1 -- 172.21.15.59:0/855103284 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f5c84057b50 con 0x7f5c90100050 2023-12-19T18:46:30.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.279+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5c78068f60 msgr2=0x7f5c7806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:30.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.279+0000 7f5c963b0700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5c78068f60 0x7f5c7806b410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5c800041d0 tx=0x7f5c8000b040 comp rx=0 tx=0).stop 2023-12-19T18:46:30.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.279+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 msgr2=0x7f5c9010fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:30.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.279+0000 7f5c963b0700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 0x7f5c9010fc70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009fd0 tx=0x7f5c84009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:30.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.279+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 shutdown_connections 2023-12-19T18:46:30.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.279+0000 7f5c963b0700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c900fd730 0x7f5c9010f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:30.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.279+0000 7f5c963b0700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f5c78068f60 0x7f5c7806b410 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:30.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.279+0000 7f5c963b0700 1 --2- 172.21.15.59:0/855103284 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5c90100050 0x7f5c9010fc70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:30.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.279+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 >> 172.21.15.59:0/855103284 conn(0x7f5c900f7190 msgr2=0x7f5c90100cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:30.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.280+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 shutdown_connections 2023-12-19T18:46:30.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:30.280+0000 7f5c963b0700 1 -- 172.21.15.59:0/855103284 wait complete. 2023-12-19T18:46:30.971 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set global log_to_journald false --force' 2023-12-19T18:46:31.268 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:31.683 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:31 smithi059 ceph-mon[80661]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2023-12-19T18:46:32.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:31 smithi089 ceph-mon[88459]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2023-12-19T18:46:32.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 -- 172.21.15.59:0/4216474591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 msgr2=0x7ff3f40fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:32.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 --2- 172.21.15.59:0/4216474591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 0x7ff3f40fe250 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc004940 tx=0x7ff3dc009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:32.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 -- 172.21.15.59:0/4216474591 shutdown_connections 2023-12-19T18:46:32.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 --2- 172.21.15.59:0/4216474591 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 0x7ff3f40fe250 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:32.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 --2- 172.21.15.59:0/4216474591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f40fcbf0 0x7ff3f40fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:32.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 -- 172.21.15.59:0/4216474591 >> 172.21.15.59:0/4216474591 conn(0x7ff3f40f82d0 msgr2=0x7ff3f40fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:32.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 -- 172.21.15.59:0/4216474591 shutdown_connections 2023-12-19T18:46:32.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 -- 172.21.15.59:0/4216474591 wait complete. 2023-12-19T18:46:32.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 Processor -- start 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.849+0000 7ff3fa778700 1 -- start start 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3fa778700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f40fcbf0 0x7ff3f410d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3fa778700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 0x7ff3f410da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3fa778700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f410e000 con 0x7ff3f40fde00 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3fa778700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f410e8a0 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3ebfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 0x7ff3f410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3f3fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f40fcbf0 0x7ff3f410d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3ebfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 0x7ff3f410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49208/0 (socket says 172.21.15.59:49208) 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3ebfff700 1 -- 172.21.15.59:0/3324449451 learned_addr learned my addr 172.21.15.59:0/3324449451 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3f3fff700 1 -- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 msgr2=0x7ff3f410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3f3fff700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 0x7ff3f410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3f3fff700 1 -- 172.21.15.59:0/3324449451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc005040 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3ebfff700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 0x7ff3f410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:46:32.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3f3fff700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f40fcbf0 0x7ff3f410d4c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40fdc60 tx=0x7ff3e400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3f17fa700 1 -- 172.21.15.59:0/3324449451 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3e4004e90 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3f17fa700 1 -- 172.21.15.59:0/3324449451 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3e400f040 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f410eb20 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.850+0000 7ff3f17fa700 1 -- 172.21.15.59:0/3324449451 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3e400c890 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.851+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f410f060 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.851+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f404bb80 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.852+0000 7ff3f17fa700 1 -- 172.21.15.59:0/3324449451 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff3e400cb00 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.852+0000 7ff3f17fa700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff3e0068f60 0x7ff3e006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.852+0000 7ff3f17fa700 1 -- 172.21.15.59:0/3324449451 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff3e4086830 con 0x7ff3f40fcbf0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.852+0000 7ff3ebfff700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff3e0068f60 0x7ff3e006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:32.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.853+0000 7ff3ebfff700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff3e0068f60 0x7ff3e006b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc0135e0 tx=0x7ff3dc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:32.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:32.857+0000 7ff3f17fa700 1 -- 172.21.15.59:0/3324449451 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff3e404da00 con 0x7ff3f40fcbf0 2023-12-19T18:46:33.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.062+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7ff3f4105590 con 0x7ff3f40fcbf0 2023-12-19T18:46:33.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.062+0000 7ff3f17fa700 1 -- 172.21.15.59:0/3324449451 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7ff3e4056770 con 0x7ff3f40fcbf0 2023-12-19T18:46:33.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff3e0068f60 msgr2=0x7ff3e006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:33.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff3e0068f60 0x7ff3e006b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc0135e0 tx=0x7ff3dc009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:33.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f40fcbf0 msgr2=0x7ff3f410d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:33.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f40fcbf0 0x7ff3f410d4c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40fdc60 tx=0x7ff3e400b040 comp rx=0 tx=0).stop 2023-12-19T18:46:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 shutdown_connections 2023-12-19T18:46:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3f40fcbf0 0x7ff3f410d4c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ff3e0068f60 0x7ff3e006b410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 --2- 172.21.15.59:0/3324449451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3f40fde00 0x7ff3f410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 >> 172.21.15.59:0/3324449451 conn(0x7ff3f40f82d0 msgr2=0x7ff3f4101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 shutdown_connections 2023-12-19T18:46:33.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:33.065+0000 7ff3fa778700 1 -- 172.21.15.59:0/3324449451 wait complete. 2023-12-19T18:46:33.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:32 smithi089 ceph-mon[88459]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2023-12-19T18:46:33.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:32 smithi059 ceph-mon[80661]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2023-12-19T18:46:33.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2023-12-19T18:46:34.051 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:35.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.738+0000 7f482b389700 1 -- 172.21.15.59:0/897964124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240ffe50 msgr2=0x7f4824102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:35.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.738+0000 7f482b389700 1 --2- 172.21.15.59:0/897964124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240ffe50 0x7f4824102230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4814004940 tx=0x7f4814009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:35.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.738+0000 7f482b389700 1 -- 172.21.15.59:0/897964124 shutdown_connections 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.738+0000 7f482b389700 1 --2- 172.21.15.59:0/897964124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240ffe50 0x7f4824102230 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.738+0000 7f482b389700 1 --2- 172.21.15.59:0/897964124 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48240fd530 0x7f48240ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.738+0000 7f482b389700 1 -- 172.21.15.59:0/897964124 >> 172.21.15.59:0/897964124 conn(0x7f48240f6f70 msgr2=0x7f48240f93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.739+0000 7f482b389700 1 -- 172.21.15.59:0/897964124 shutdown_connections 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.739+0000 7f482b389700 1 -- 172.21.15.59:0/897964124 wait complete. 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.739+0000 7f482b389700 1 Processor -- start 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.739+0000 7f482b389700 1 -- start start 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.739+0000 7f482b389700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48240fd530 0x7f48240fcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.739+0000 7f482b389700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240ffe50 0x7f48240fb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.739+0000 7f482b389700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48240fba40 con 0x7f48240fd530 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.739+0000 7f482b389700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48240fbbb0 con 0x7f48240ffe50 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.739+0000 7f482a387700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48240fd530 0x7f48240fcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:35.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f482a387700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48240fd530 0x7f48240fcfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49222/0 (socket says 172.21.15.59:49222) 2023-12-19T18:46:35.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f482a387700 1 -- 172.21.15.59:0/1028634899 learned_addr learned my addr 172.21.15.59:0/1028634899 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:35.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f4829b86700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240ffe50 0x7f48240fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:35.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f482a387700 1 -- 172.21.15.59:0/1028634899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240ffe50 msgr2=0x7f48240fb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:35.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f482a387700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240ffe50 0x7f48240fb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:35.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f482a387700 1 -- 172.21.15.59:0/1028634899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820005040 con 0x7f48240fd530 2023-12-19T18:46:35.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f482a387700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48240fd530 0x7f48240fcfc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f48200096d0 tx=0x7f4820009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:35.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f481affd700 1 -- 172.21.15.59:0/1028634899 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f482000e840 con 0x7f48240fd530 2023-12-19T18:46:35.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4814005040 con 0x7f48240fd530 2023-12-19T18:46:35.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f481affd700 1 -- 172.21.15.59:0/1028634899 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f482000f040 con 0x7f48240fd530 2023-12-19T18:46:35.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.740+0000 7f481affd700 1 -- 172.21.15.59:0/1028634899 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4820013d30 con 0x7f48240fd530 2023-12-19T18:46:35.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.741+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48240fc260 con 0x7f48240fd530 2023-12-19T18:46:35.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.742+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48241073c0 con 0x7f48240fd530 2023-12-19T18:46:35.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.742+0000 7f481affd700 1 -- 172.21.15.59:0/1028634899 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f482000e9a0 con 0x7f48240fd530 2023-12-19T18:46:35.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.743+0000 7f481affd700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4810068f60 0x7f481006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:35.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.743+0000 7f481affd700 1 -- 172.21.15.59:0/1028634899 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4820086b30 con 0x7f48240fd530 2023-12-19T18:46:35.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.743+0000 7f4829b86700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4810068f60 0x7f481006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:35.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.744+0000 7f4829b86700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4810068f60 0x7f481006b410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4814000c00 tx=0x7f4814009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:35.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.748+0000 7f481affd700 1 -- 172.21.15.59:0/1028634899 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4820056a40 con 0x7f48240fd530 2023-12-19T18:46:35.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:35 smithi059 ceph-mon[80661]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:46:35.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.978+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "target": ["mon-mgr", ""]}) v1 -- 0x7f482402b1f0 con 0x7f4810068f60 2023-12-19T18:46:35.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.983+0000 7f481affd700 1 -- 172.21.15.59:0/1028634899 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f482402b1f0 con 0x7f4810068f60 2023-12-19T18:46:35.984 INFO:teuthology.orchestra.run.smithi059.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:46:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4810068f60 msgr2=0x7f481006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4810068f60 0x7f481006b410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4814000c00 tx=0x7f4814009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48240fd530 msgr2=0x7f48240fcfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48240fd530 0x7f48240fcfc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f48200096d0 tx=0x7f4820009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 shutdown_connections 2023-12-19T18:46:35.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4810068f60 0x7f481006b410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:35.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f48240fd530 0x7f48240fcfc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:35.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 --2- 172.21.15.59:0/1028634899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f48240ffe50 0x7f48240fb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:35.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 >> 172.21.15.59:0/1028634899 conn(0x7f48240f6f70 msgr2=0x7f48240f93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:35.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 shutdown_connections 2023-12-19T18:46:35.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:35.986+0000 7f482b389700 1 -- 172.21.15.59:0/1028634899 wait complete. 2023-12-19T18:46:36.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:35 smithi089 ceph-mon[88459]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:46:36.681 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2023-12-19T18:46:36.682 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T18:46:36.682 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2023-12-19T18:46:36.995 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:46:37.250 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:36 smithi059 ceph-mon[80661]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:37.250 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:36 smithi059 ceph-mon[80661]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:46:37.250 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:36 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:37.251 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:36 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:46:37.251 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:36 smithi059 ceph-mon[80661]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:46:37.251 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:36 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:46:37.251 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:36 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:46:37.251 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:36 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:37.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:36 smithi089 ceph-mon[88459]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:37.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:36 smithi089 ceph-mon[88459]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:46:37.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:36 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:37.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:36 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:46:37.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:36 smithi089 ceph-mon[88459]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:46:37.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:36 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:46:37.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:36 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T18:46:37.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:36 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:46:38.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:37 smithi059 ceph-mon[80661]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:46:38.156 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:37 smithi089 ceph-mon[88459]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:46:38.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 -- 172.21.15.59:0/777197440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f480fd730 msgr2=0x7f4f480ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:38.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 --2- 172.21.15.59:0/777197440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f480fd730 0x7f4f480ffb10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4f44004940 tx=0x7f4f44009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 -- 172.21.15.59:0/777197440 shutdown_connections 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 --2- 172.21.15.59:0/777197440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f48100050 0x7f4f48102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 --2- 172.21.15.59:0/777197440 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f480fd730 0x7f4f480ffb10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 -- 172.21.15.59:0/777197440 >> 172.21.15.59:0/777197440 conn(0x7f4f480f7190 msgr2=0x7f4f480f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 -- 172.21.15.59:0/777197440 shutdown_connections 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 -- 172.21.15.59:0/777197440 wait complete. 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 Processor -- start 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.790+0000 7f4f5001e700 1 -- start start 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f5001e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f480fd730 0x7f4f4810d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f5001e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f48100050 0x7f4f4810da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f5001e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4810e040 con 0x7f4f48100050 2023-12-19T18:46:38.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f5001e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4810e8e0 con 0x7f4f480fd730 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f4d5b9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f48100050 0x7f4f4810da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f4d5b9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f48100050 0x7f4f4810da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49240/0 (socket says 172.21.15.59:49240) 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f4d5b9700 1 -- 172.21.15.59:0/3278612833 learned_addr learned my addr 172.21.15.59:0/3278612833 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f4ddba700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f480fd730 0x7f4f4810d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f4d5b9700 1 -- 172.21.15.59:0/3278612833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f480fd730 msgr2=0x7f4f4810d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f4d5b9700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f480fd730 0x7f4f4810d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f4d5b9700 1 -- 172.21.15.59:0/3278612833 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44005040 con 0x7f4f48100050 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.791+0000 7f4f4d5b9700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f48100050 0x7f4f4810da40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4f38002a50 tx=0x7f4f380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.792+0000 7f4f3e7fc700 1 -- 172.21.15.59:0/3278612833 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f38007cc0 con 0x7f4f48100050 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.792+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4810ebc0 con 0x7f4f48100050 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.792+0000 7f4f3e7fc700 1 -- 172.21.15.59:0/3278612833 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f3800f040 con 0x7f4f48100050 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.792+0000 7f4f3e7fc700 1 -- 172.21.15.59:0/3278612833 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f3800d9c0 con 0x7f4f48100050 2023-12-19T18:46:38.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.792+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4810f100 con 0x7f4f48100050 2023-12-19T18:46:38.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.793+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c0052e0 con 0x7f4f48100050 2023-12-19T18:46:38.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.793+0000 7f4f3e7fc700 1 -- 172.21.15.59:0/3278612833 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4f3800db20 con 0x7f4f48100050 2023-12-19T18:46:38.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.794+0000 7f4f3e7fc700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f34068f70 0x7f4f3406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:38.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.794+0000 7f4f3e7fc700 1 -- 172.21.15.59:0/3278612833 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4f38086a40 con 0x7f4f48100050 2023-12-19T18:46:38.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.794+0000 7f4f4ddba700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f34068f70 0x7f4f3406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:38.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.794+0000 7f4f4ddba700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f34068f70 0x7f4f3406b420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400b3b0 tx=0x7f4f44009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:38.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:38.798+0000 7f4f3e7fc700 1 -- 172.21.15.59:0/3278612833 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4f38056950 con 0x7f4f48100050 2023-12-19T18:46:39.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.068+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f2c000bf0 con 0x7f4f34068f70 2023-12-19T18:46:39.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:38 smithi059 ceph-mon[80661]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:46:39.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:46:39.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.069+0000 7f4f3e7fc700 1 -- 172.21.15.59:0/3278612833 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f4f2c000bf0 con 0x7f4f34068f70 2023-12-19T18:46:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f34068f70 msgr2=0x7f4f3406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f34068f70 0x7f4f3406b420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400b3b0 tx=0x7f4f44009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f48100050 msgr2=0x7f4f4810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f48100050 0x7f4f4810da40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4f38002a50 tx=0x7f4f380046b0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 shutdown_connections 2023-12-19T18:46:39.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f480fd730 0x7f4f4810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f4f34068f70 0x7f4f3406b420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 --2- 172.21.15.59:0/3278612833 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4f48100050 0x7f4f4810da40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 >> 172.21.15.59:0/3278612833 conn(0x7f4f480f7190 msgr2=0x7f4f480f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:39.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 shutdown_connections 2023-12-19T18:46:39.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.072+0000 7f4f5001e700 1 -- 172.21.15.59:0/3278612833 wait complete. 2023-12-19T18:46:39.084 INFO:teuthology.orchestra.run.smithi059.stdout:true 2023-12-19T18:46:39.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.196+0000 7f93f420f700 1 -- 172.21.15.59:0/3304722951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec0ffd70 msgr2=0x7f93ec1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.196+0000 7f93f420f700 1 --2- 172.21.15.59:0/3304722951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec0ffd70 0x7f93ec1021d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f93dc004940 tx=0x7f93dc009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:39.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.196+0000 7f93f420f700 1 -- 172.21.15.59:0/3304722951 shutdown_connections 2023-12-19T18:46:39.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.196+0000 7f93f420f700 1 --2- 172.21.15.59:0/3304722951 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec0ffd70 0x7f93ec1021d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.196+0000 7f93f420f700 1 --2- 172.21.15.59:0/3304722951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93ec066090 0x7f93ec0ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.196+0000 7f93f420f700 1 -- 172.21.15.59:0/3304722951 >> 172.21.15.59:0/3304722951 conn(0x7f93ec0f7190 msgr2=0x7f93ec0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:39.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.196+0000 7f93f420f700 1 -- 172.21.15.59:0/3304722951 shutdown_connections 2023-12-19T18:46:39.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.196+0000 7f93f420f700 1 -- 172.21.15.59:0/3304722951 wait complete. 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.197+0000 7f93f420f700 1 Processor -- start 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.197+0000 7f93f420f700 1 -- start start 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.197+0000 7f93f420f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec066090 0x7f93ec10d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.197+0000 7f93f420f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93ec0ffd70 0x7f93ec10d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.197+0000 7f93f420f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec10dfb0 con 0x7f93ec066090 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.197+0000 7f93f420f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec10e850 con 0x7f93ec0ffd70 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.197+0000 7f93f1fab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec066090 0x7f93ec10d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.197+0000 7f93f1fab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec066090 0x7f93ec10d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49268/0 (socket says 172.21.15.59:49268) 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93f1fab700 1 -- 172.21.15.59:0/2337347274 learned_addr learned my addr 172.21.15.59:0/2337347274 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93f17aa700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93ec0ffd70 0x7f93ec10d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93f1fab700 1 -- 172.21.15.59:0/2337347274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93ec0ffd70 msgr2=0x7f93ec10d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93f1fab700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93ec0ffd70 0x7f93ec10d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93f1fab700 1 -- 172.21.15.59:0/2337347274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc005040 con 0x7f93ec066090 2023-12-19T18:46:39.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93f1fab700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec066090 0x7f93ec10d470 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f93e8002a80 tx=0x7f93e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:39.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93e27fc700 1 -- 172.21.15.59:0/2337347274 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e8007cc0 con 0x7f93ec066090 2023-12-19T18:46:39.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93e27fc700 1 -- 172.21.15.59:0/2337347274 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93e800f040 con 0x7f93ec066090 2023-12-19T18:46:39.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec10eb30 con 0x7f93ec066090 2023-12-19T18:46:39.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93e27fc700 1 -- 172.21.15.59:0/2337347274 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e800da60 con 0x7f93ec066090 2023-12-19T18:46:39.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.198+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec10f070 con 0x7f93ec066090 2023-12-19T18:46:39.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.199+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec04bb80 con 0x7f93ec066090 2023-12-19T18:46:39.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.200+0000 7f93e27fc700 1 -- 172.21.15.59:0/2337347274 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f93e800dbc0 con 0x7f93ec066090 2023-12-19T18:46:39.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.201+0000 7f93e27fc700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f93d8068f70 0x7f93d806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:39.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.201+0000 7f93e27fc700 1 -- 172.21.15.59:0/2337347274 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f93e80868d0 con 0x7f93ec066090 2023-12-19T18:46:39.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.201+0000 7f93f17aa700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f93d8068f70 0x7f93d806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:39.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.201+0000 7f93f17aa700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f93d8068f70 0x7f93d806b420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f93dc004890 tx=0x7f93dc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:39.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.206+0000 7f93e27fc700 1 -- 172.21.15.59:0/2337347274 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f93e80567e0 con 0x7f93ec066090 2023-12-19T18:46:39.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:38 smithi089 ceph-mon[88459]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:46:39.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:46:39.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.486+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f93ec100a10 con 0x7f93d8068f70 2023-12-19T18:46:39.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.487+0000 7f93e27fc700 1 -- 172.21.15.59:0/2337347274 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f93ec100a10 con 0x7f93d8068f70 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.489+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f93d8068f70 msgr2=0x7f93d806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.489+0000 7f93f420f700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f93d8068f70 0x7f93d806b420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f93dc004890 tx=0x7f93dc009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.489+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec066090 msgr2=0x7f93ec10d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.489+0000 7f93f420f700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec066090 0x7f93ec10d470 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f93e8002a80 tx=0x7f93e80046b0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.489+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 shutdown_connections 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.489+0000 7f93f420f700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f93d8068f70 0x7f93d806b420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.489+0000 7f93f420f700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f93ec066090 0x7f93ec10d470 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.489+0000 7f93f420f700 1 --2- 172.21.15.59:0/2337347274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93ec0ffd70 0x7f93ec10d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.489+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 >> 172.21.15.59:0/2337347274 conn(0x7f93ec0f7190 msgr2=0x7f93ec0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:39.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.490+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 shutdown_connections 2023-12-19T18:46:39.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.490+0000 7f93f420f700 1 -- 172.21.15.59:0/2337347274 wait complete. 2023-12-19T18:46:39.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.615+0000 7ffa33e87700 1 -- 172.21.15.59:0/1247583491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fc4a0 msgr2=0x7ffa2c0fc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.615+0000 7ffa33e87700 1 --2- 172.21.15.59:0/1247583491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fc4a0 0x7ffa2c0fc910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c004940 tx=0x7ffa1c009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:39.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.615+0000 7ffa33e87700 1 -- 172.21.15.59:0/1247583491 shutdown_connections 2023-12-19T18:46:39.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.615+0000 7ffa33e87700 1 --2- 172.21.15.59:0/1247583491 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fc4a0 0x7ffa2c0fc910 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.615+0000 7ffa33e87700 1 --2- 172.21.15.59:0/1247583491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0fb470 0x7ffa2c0fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.615+0000 7ffa33e87700 1 -- 172.21.15.59:0/1247583491 >> 172.21.15.59:0/1247583491 conn(0x7ffa2c0f64f0 msgr2=0x7ffa2c0f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.615+0000 7ffa33e87700 1 -- 172.21.15.59:0/1247583491 shutdown_connections 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.615+0000 7ffa33e87700 1 -- 172.21.15.59:0/1247583491 wait complete. 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa33e87700 1 Processor -- start 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa33e87700 1 -- start start 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa33e87700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fb470 0x7ffa2c10cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa33e87700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0fc4a0 0x7ffa2c10af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa33e87700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c10b590 con 0x7ffa2c0fb470 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa33e87700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c10b700 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa31c23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fb470 0x7ffa2c10cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa31422700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0fc4a0 0x7ffa2c10af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:39.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa31c23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fb470 0x7ffa2c10cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49280/0 (socket says 172.21.15.59:49280) 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa31c23700 1 -- 172.21.15.59:0/2742902912 learned_addr learned my addr 172.21.15.59:0/2742902912 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.616+0000 7ffa31422700 1 -- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fb470 msgr2=0x7ffa2c10cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.617+0000 7ffa31422700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fb470 0x7ffa2c10cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.617+0000 7ffa31422700 1 -- 172.21.15.59:0/2742902912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c005040 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.617+0000 7ffa31c23700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fb470 0x7ffa2c10cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.617+0000 7ffa31422700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0fc4a0 0x7ffa2c10af90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00b0d0 tx=0x7ffa1c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.617+0000 7ffa227fc700 1 -- 172.21.15.59:0/2742902912 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1c00e070 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.617+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c10b980 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.617+0000 7ffa227fc700 1 -- 172.21.15.59:0/2742902912 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa1c01c400 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.617+0000 7ffa227fc700 1 -- 172.21.15.59:0/2742902912 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1c00b650 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.617+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c115510 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.618+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c04bb80 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.618+0000 7ffa227fc700 1 -- 172.21.15.59:0/2742902912 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ffa1c01c9e0 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.619+0000 7ffa227fc700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffa18068f60 0x7ffa1806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:39.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.619+0000 7ffa227fc700 1 -- 172.21.15.59:0/2742902912 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ffa1c0880a0 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.619+0000 7ffa31c23700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffa18068f60 0x7ffa1806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:39.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.619+0000 7ffa31c23700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffa18068f60 0x7ffa1806b410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c10c8a0 tx=0x7ffa2800c040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:39.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.623+0000 7ffa227fc700 1 -- 172.21.15.59:0/2742902912 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ffa1c057f90 con 0x7ffa2c0fc4a0 2023-12-19T18:46:39.899 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.898+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa2c100df0 con 0x7ffa18068f60 2023-12-19T18:46:39.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.910+0000 7ffa227fc700 1 -- 172.21.15.59:0/2742902912 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7ffa2c100df0 con 0x7ffa18068f60 2023-12-19T18:46:39.911 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-19T18:46:39.911 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (4m) 24s ago 6m 25.1M - 0.25.0 c8568f914cd2 528f56ce4fdd 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (6m) 24s ago 6m 6660k - 17.2.7-131-ga88f3edf f9ecdbaa1030 03ae34e0a398 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi089 smithi089 running (5m) 31s ago 5m 6665k - 17.2.7-131-ga88f3edf f9ecdbaa1030 8a39e517426d 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (4m) 24s ago 5m 87.0M - 9.4.7 954c08fa6188 a19de18455de 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.abqklt smithi059 running (50s) 24s ago 50s 10.4M - 17.2.7-131-ga88f3edf f9ecdbaa1030 c303f5a4640b 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.rtqzmc smithi059 running (44s) 24s ago 43s 14.2M - 17.2.7-131-ga88f3edf f9ecdbaa1030 a2c3ca2a9ed2 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi089.dfrkla smithi089 running (41s) 31s ago 41s 14.5M - 17.2.7-131-ga88f3edf f9ecdbaa1030 f707fc4d33e4 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi089.dpgsbt smithi089 running (48s) 31s ago 47s 12.1M - 17.2.7-131-ga88f3edf f9ecdbaa1030 ce89c8fa8ca4 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.ccbnob smithi059 *:9283 running (7m) 24s ago 7m 479M - 17.2.7-131-ga88f3edf f9ecdbaa1030 9ce46f56adc1 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi089.isaaav smithi089 *:8443,9283 running (5m) 31s ago 5m 434M - 17.2.7-131-ga88f3edf f9ecdbaa1030 5890af67d417 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (7m) 24s ago 7m 45.1M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 297c7051711f 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi089 smithi089 running (5m) 31s ago 5m 33.0M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 5ac855c460fa 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (6m) 24s ago 6m 16.0M - 1.5.0 0da6a335fe13 bfe5bff75fbd 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi089 smithi089 *:9100 running (5m) 31s ago 5m 15.7M - 1.5.0 0da6a335fe13 057c3879e7a2 2023-12-19T18:46:39.912 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (4m) 24s ago 4m 48.2M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 87e75bce1c5b 2023-12-19T18:46:39.913 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (3m) 24s ago 3m 48.5M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 1b133ef58db0 2023-12-19T18:46:39.913 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (3m) 24s ago 3m 45.0M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 0f56936ea44c 2023-12-19T18:46:39.913 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi089 running (3m) 31s ago 3m 44.7M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 3eafaf27a901 2023-12-19T18:46:39.913 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi089 running (2m) 31s ago 2m 44.0M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 cc630810bb79 2023-12-19T18:46:39.913 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi089 running (2m) 31s ago 2m 41.3M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 f1d513a4069b 2023-12-19T18:46:39.913 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (4m) 24s ago 5m 52.1M - 2.43.0 a07b618ecd1d 5884a00b2f95 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffa18068f60 msgr2=0x7ffa1806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffa18068f60 0x7ffa1806b410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c10c8a0 tx=0x7ffa2800c040 comp rx=0 tx=0).stop 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0fc4a0 msgr2=0x7ffa2c10af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0fc4a0 0x7ffa2c10af90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00b0d0 tx=0x7ffa1c009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 shutdown_connections 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7ffa18068f60 0x7ffa1806b410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffa2c0fb470 0x7ffa2c10cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 --2- 172.21.15.59:0/2742902912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa2c0fc4a0 0x7ffa2c10af90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 >> 172.21.15.59:0/2742902912 conn(0x7ffa2c0f64f0 msgr2=0x7ffa2c0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 shutdown_connections 2023-12-19T18:46:39.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:39.913+0000 7ffa33e87700 1 -- 172.21.15.59:0/2742902912 wait complete. 2023-12-19T18:46:40.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.056+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/1809232375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 msgr2=0x7fb7b40fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:40.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.056+0000 7fb7ba1e4700 1 --2- 172.21.15.59:0/1809232375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 0x7fb7b40fd000 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4004940 tx=0x7fb7a4009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:40.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.056+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/1809232375 shutdown_connections 2023-12-19T18:46:40.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.056+0000 7fb7ba1e4700 1 --2- 172.21.15.59:0/1809232375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b40fde00 0x7fb7b40fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.056+0000 7fb7ba1e4700 1 --2- 172.21.15.59:0/1809232375 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 0x7fb7b40fd000 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.056+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/1809232375 >> 172.21.15.59:0/1809232375 conn(0x7fb7b40f82d0 msgr2=0x7fb7b40fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:40.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.056+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/1809232375 shutdown_connections 2023-12-19T18:46:40.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.056+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/1809232375 wait complete. 2023-12-19T18:46:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7ba1e4700 1 Processor -- start 2023-12-19T18:46:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7ba1e4700 1 -- start start 2023-12-19T18:46:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7ba1e4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 0x7fb7b410f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7ba1e4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b40fde00 0x7fb7b410fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7ba1e4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b4110200 con 0x7fb7b40fcbf0 2023-12-19T18:46:40.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7ba1e4700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b4110aa0 con 0x7fb7b40fde00 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7b37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 0x7fb7b410f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7b2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b40fde00 0x7fb7b410fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7b37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 0x7fb7b410f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49294/0 (socket says 172.21.15.59:49294) 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7b37fe700 1 -- 172.21.15.59:0/175034862 learned_addr learned my addr 172.21.15.59:0/175034862 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.057+0000 7fb7b2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b40fde00 0x7fb7b410fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.59:32986/0 (socket says 172.21.15.59:32986) 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb7b2ffd700 1 -- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 msgr2=0x7fb7b410f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb7b2ffd700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 0x7fb7b410f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb7b2ffd700 1 -- 172.21.15.59:0/175034862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a4005040 con 0x7fb7b40fde00 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb7b37fe700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 0x7fb7b410f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb7b2ffd700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b40fde00 0x7fb7b410fc00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009fd0 tx=0x7fb7a800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb79bfff700 1 -- 172.21.15.59:0/175034862 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7a8004e90 con 0x7fb7b40fde00 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b4110d20 con 0x7fb7b40fde00 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb79bfff700 1 -- 172.21.15.59:0/175034862 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7a800f040 con 0x7fb7b40fde00 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb79bfff700 1 -- 172.21.15.59:0/175034862 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7a800c9b0 con 0x7fb7b40fde00 2023-12-19T18:46:40.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.058+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b4111260 con 0x7fb7b40fde00 2023-12-19T18:46:40.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.060+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b4101ea0 con 0x7fb7b40fde00 2023-12-19T18:46:40.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.060+0000 7fb79bfff700 1 -- 172.21.15.59:0/175034862 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb7a800cb10 con 0x7fb7b40fde00 2023-12-19T18:46:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.061+0000 7fb79bfff700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb79c068f60 0x7fb79c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.061+0000 7fb79bfff700 1 -- 172.21.15.59:0/175034862 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb7a80867f0 con 0x7fb7b40fde00 2023-12-19T18:46:40.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.061+0000 7fb7b37fe700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb79c068f60 0x7fb79c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:40.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.061+0000 7fb7b37fe700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb79c068f60 0x7fb79c06b410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4013600 tx=0x7fb7a4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:40.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.065+0000 7fb79bfff700 1 -- 172.21.15.59:0/175034862 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb7a804da00 con 0x7fb7b40fde00 2023-12-19T18:46:40.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:39 smithi059 ceph-mon[80661]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:40.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:39 smithi059 ceph-mon[80661]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:40.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:39 smithi089 ceph-mon[88459]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:40.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:39 smithi089 ceph-mon[88459]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:40.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.433+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb7b404bb80 con 0x7fb7b40fde00 2023-12-19T18:46:40.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.433+0000 7fb79bfff700 1 -- 172.21.15.59:0/175034862 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fb7a8086070 con 0x7fb7b40fde00 2023-12-19T18:46:40.434 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": { 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 2 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": { 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 2 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": { 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 6 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": { 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 4 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "overall": { 2023-12-19T18:46:40.435 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 14 2023-12-19T18:46:40.436 INFO:teuthology.orchestra.run.smithi059.stdout: } 2023-12-19T18:46:40.436 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-12-19T18:46:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.435+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb79c068f60 msgr2=0x7fb79c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.435+0000 7fb7ba1e4700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb79c068f60 0x7fb79c06b410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4013600 tx=0x7fb7a4009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.436+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b40fde00 msgr2=0x7fb7b410fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.436+0000 7fb7ba1e4700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b40fde00 0x7fb7b410fc00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009fd0 tx=0x7fb7a800b040 comp rx=0 tx=0).stop 2023-12-19T18:46:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.436+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 shutdown_connections 2023-12-19T18:46:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.436+0000 7fb7ba1e4700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fb79c068f60 0x7fb79c06b410 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.436+0000 7fb7ba1e4700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7b40fcbf0 0x7fb7b410f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.436+0000 7fb7ba1e4700 1 --2- 172.21.15.59:0/175034862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7b40fde00 0x7fb7b410fc00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.436+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 >> 172.21.15.59:0/175034862 conn(0x7fb7b40f82d0 msgr2=0x7fb7b40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.436+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 shutdown_connections 2023-12-19T18:46:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.436+0000 7fb7ba1e4700 1 -- 172.21.15.59:0/175034862 wait complete. 2023-12-19T18:46:40.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.556+0000 7fcfda18d700 1 -- 172.21.15.59:0/9469891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd4066920 msgr2=0x7fcfd40ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:40.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.556+0000 7fcfda18d700 1 --2- 172.21.15.59:0/9469891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd4066920 0x7fcfd40ff7a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4004940 tx=0x7fcfc4009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:40.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.556+0000 7fcfda18d700 1 -- 172.21.15.59:0/9469891 shutdown_connections 2023-12-19T18:46:40.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.556+0000 7fcfda18d700 1 --2- 172.21.15.59:0/9469891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd40ffd70 0x7fcfd41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.556+0000 7fcfda18d700 1 --2- 172.21.15.59:0/9469891 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd4066920 0x7fcfd40ff7a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.556+0000 7fcfda18d700 1 -- 172.21.15.59:0/9469891 >> 172.21.15.59:0/9469891 conn(0x7fcfd40f71d0 msgr2=0x7fcfd40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.556+0000 7fcfda18d700 1 -- 172.21.15.59:0/9469891 shutdown_connections 2023-12-19T18:46:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.556+0000 7fcfda18d700 1 -- 172.21.15.59:0/9469891 wait complete. 2023-12-19T18:46:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.557+0000 7fcfda18d700 1 Processor -- start 2023-12-19T18:46:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.557+0000 7fcfda18d700 1 -- start start 2023-12-19T18:46:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfda18d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd4066920 0x7fcfd410d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfda18d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd40ffd70 0x7fcfd410d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfda18d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd410dff0 con 0x7fcfd4066920 2023-12-19T18:46:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfda18d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd410e890 con 0x7fcfd40ffd70 2023-12-19T18:46:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfd37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd4066920 0x7fcfd410d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfd2ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd40ffd70 0x7fcfd410d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:40.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfd37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd4066920 0x7fcfd410d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49312/0 (socket says 172.21.15.59:49312) 2023-12-19T18:46:40.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfd37fe700 1 -- 172.21.15.59:0/3235344994 learned_addr learned my addr 172.21.15.59:0/3235344994 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:40.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfd2ffd700 1 -- 172.21.15.59:0/3235344994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd4066920 msgr2=0x7fcfd410d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:40.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfd2ffd700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd4066920 0x7fcfd410d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfd2ffd700 1 -- 172.21.15.59:0/3235344994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc4005040 con 0x7fcfd40ffd70 2023-12-19T18:46:40.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfd2ffd700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd40ffd70 0x7fcfd410d9f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4066660 tx=0x7fcfc800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:40.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.558+0000 7fcfbbfff700 1 -- 172.21.15.59:0/3235344994 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfc8004e90 con 0x7fcfd40ffd70 2023-12-19T18:46:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.559+0000 7fcfbbfff700 1 -- 172.21.15.59:0/3235344994 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfc800f040 con 0x7fcfd40ffd70 2023-12-19T18:46:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.559+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd410eb10 con 0x7fcfd40ffd70 2023-12-19T18:46:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.559+0000 7fcfbbfff700 1 -- 172.21.15.59:0/3235344994 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfc800c980 con 0x7fcfd40ffd70 2023-12-19T18:46:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.559+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd410f050 con 0x7fcfd40ffd70 2023-12-19T18:46:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.560+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd404bb80 con 0x7fcfd40ffd70 2023-12-19T18:46:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.560+0000 7fcfbbfff700 1 -- 172.21.15.59:0/3235344994 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcfc800cae0 con 0x7fcfd40ffd70 2023-12-19T18:46:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.560+0000 7fcfbbfff700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcfbc068f60 0x7fcfbc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.561+0000 7fcfbbfff700 1 -- 172.21.15.59:0/3235344994 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcfc80867a0 con 0x7fcfd40ffd70 2023-12-19T18:46:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.561+0000 7fcfd37fe700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcfbc068f60 0x7fcfbc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.561+0000 7fcfd37fe700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcfbc068f60 0x7fcfbc06b410 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40135e0 tx=0x7fcfc4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:40.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.565+0000 7fcfbbfff700 1 -- 172.21.15.59:0/3235344994 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcfc804da00 con 0x7fcfd40ffd70 2023-12-19T18:46:40.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.863+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcfd41055a0 con 0x7fcfd40ffd70 2023-12-19T18:46:40.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.863+0000 7fcfbbfff700 1 -- 172.21.15.59:0/3235344994 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1805 (secure 0 0 0) 0x7fcfc80566c0 con 0x7fcfd40ffd70 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:e15 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 15 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:created 2023-12-19T18:45:45.631056+0000 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2023-12-19T18:46:03.615908+0000 2023-12-19T18:46:40.865 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 47 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:in 0 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:up {0=14474} 2023-12-19T18:46:40.866 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data enabled 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 1 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.rtqzmc{0:14474} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:Standby daemons: 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:46:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.abqklt{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.59:6826/1872103080,v1:172.21.15.59:6827/1872103080] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:46:40.868 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi089.dpgsbt{-1:24291} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.89:6824/1622105230,v1:172.21.15.89:6825/1622105230] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:46:40.868 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi089.dfrkla{-1:24301} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.89:6826/3691364030,v1:172.21.15.89:6827/3691364030] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:46:40.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcfbc068f60 msgr2=0x7fcfbc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcfbc068f60 0x7fcfbc06b410 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40135e0 tx=0x7fcfc4009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd40ffd70 msgr2=0x7fcfd410d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd40ffd70 0x7fcfd410d9f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4066660 tx=0x7fcfc800b040 comp rx=0 tx=0).stop 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 shutdown_connections 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fcfbc068f60 0x7fcfbc06b410 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcfd4066920 0x7fcfd410d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 --2- 172.21.15.59:0/3235344994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd40ffd70 0x7fcfd410d9f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 >> 172.21.15.59:0/3235344994 conn(0x7fcfd40f71d0 msgr2=0x7fcfd40f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 shutdown_connections 2023-12-19T18:46:40.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.866+0000 7fcfda18d700 1 -- 172.21.15.59:0/3235344994 wait complete. 2023-12-19T18:46:40.870 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 15 2023-12-19T18:46:40.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.995+0000 7f21e3230700 1 -- 172.21.15.59:0/4285173594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0fdf60 msgr2=0x7f21dc0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:40.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.995+0000 7f21e3230700 1 --2- 172.21.15.59:0/4285173594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0fdf60 0x7f21dc0fe3b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f21c8004940 tx=0x7f21c8009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:40.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.996+0000 7f21e3230700 1 -- 172.21.15.59:0/4285173594 shutdown_connections 2023-12-19T18:46:40.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.996+0000 7f21e3230700 1 --2- 172.21.15.59:0/4285173594 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0fdf60 0x7f21dc0fe3b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.996+0000 7f21e3230700 1 --2- 172.21.15.59:0/4285173594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21dc0faed0 0x7f21dc0fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:40.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.996+0000 7f21e3230700 1 -- 172.21.15.59:0/4285173594 >> 172.21.15.59:0/4285173594 conn(0x7f21dc0f6950 msgr2=0x7f21dc0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:40.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.996+0000 7f21e3230700 1 -- 172.21.15.59:0/4285173594 shutdown_connections 2023-12-19T18:46:40.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.996+0000 7f21e3230700 1 -- 172.21.15.59:0/4285173594 wait complete. 2023-12-19T18:46:40.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.997+0000 7f21e3230700 1 Processor -- start 2023-12-19T18:46:40.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.997+0000 7f21e3230700 1 -- start start 2023-12-19T18:46:40.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.997+0000 7f21e3230700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0faed0 0x7f21dc0fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:40.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.997+0000 7f21e3230700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21dc0fdf60 0x7f21dc0fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:40.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.997+0000 7f21e3230700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0fc470 con 0x7f21dc0faed0 2023-12-19T18:46:40.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.997+0000 7f21e3230700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21dc0fc5e0 con 0x7f21dc0fdf60 2023-12-19T18:46:40.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.998+0000 7f21e0fcc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0faed0 0x7f21dc0fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:40.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.998+0000 7f21e0fcc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0faed0 0x7f21dc0fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49336/0 (socket says 172.21.15.59:49336) 2023-12-19T18:46:40.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.998+0000 7f21e0fcc700 1 -- 172.21.15.59:0/4057031556 learned_addr learned my addr 172.21.15.59:0/4057031556 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.998+0000 7f21dbfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21dc0fdf60 0x7f21dc0fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.998+0000 7f21e0fcc700 1 -- 172.21.15.59:0/4057031556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21dc0fdf60 msgr2=0x7f21dc0fbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.998+0000 7f21e0fcc700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21dc0fdf60 0x7f21dc0fbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.998+0000 7f21e0fcc700 1 -- 172.21.15.59:0/4057031556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c8005040 con 0x7f21dc0faed0 2023-12-19T18:46:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.998+0000 7f21dbfff700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21dc0fdf60 0x7f21dc0fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:46:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:40.998+0000 7f21e0fcc700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0faed0 0x7f21dc0fd9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f21d000eb70 tx=0x7f21d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.000+0000 7f21d97fa700 1 -- 172.21.15.59:0/4057031556 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d0007cc0 con 0x7f21dc0faed0 2023-12-19T18:46:41.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.000+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21dc0fc860 con 0x7f21dc0faed0 2023-12-19T18:46:41.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.000+0000 7f21d97fa700 1 -- 172.21.15.59:0/4057031556 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21d000f040 con 0x7f21dc0faed0 2023-12-19T18:46:41.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.000+0000 7f21d97fa700 1 -- 172.21.15.59:0/4057031556 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d000c9d0 con 0x7f21dc0faed0 2023-12-19T18:46:41.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.000+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21dc115580 con 0x7f21dc0faed0 2023-12-19T18:46:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.001+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21dc04bb80 con 0x7f21dc0faed0 2023-12-19T18:46:41.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.001+0000 7f21d97fa700 1 -- 172.21.15.59:0/4057031556 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f21d000cc40 con 0x7f21dc0faed0 2023-12-19T18:46:41.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.002+0000 7f21d97fa700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f21cc068f70 0x7f21cc06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:41.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.002+0000 7f21dbfff700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f21cc068f70 0x7f21cc06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:41.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.003+0000 7f21dbfff700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f21cc068f70 0x7f21cc06b420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f21c800b0d0 tx=0x7f21c8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:41.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.004+0000 7f21d97fa700 1 -- 172.21.15.59:0/4057031556 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f21d0059bb0 con 0x7f21dc0faed0 2023-12-19T18:46:41.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.008+0000 7f21d97fa700 1 -- 172.21.15.59:0/4057031556 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f21d004da00 con 0x7f21dc0faed0 2023-12-19T18:46:41.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:41 smithi059 ceph-mon[80661]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:41.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:41 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/175034862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-19T18:46:41.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:41 smithi059 ceph-mon[80661]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T18:46:41.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:41 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/3235344994' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:46:41.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:41 smithi089 ceph-mon[88459]: from='client.24369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:41.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:41 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/175034862' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-19T18:46:41.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:41 smithi089 ceph-mon[88459]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T18:46:41.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:41 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/3235344994' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:46:41.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.289+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21dc1028b0 con 0x7f21cc068f70 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.290+0000 7f21d97fa700 1 -- 172.21.15.59:0/4057031556 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f21dc1028b0 con 0x7f21cc068f70 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stdout: "in_progress": true, 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stdout: "services_complete": [], 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stdout: "progress": "", 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 image", 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stdout: "is_paused": false 2023-12-19T18:46:41.294 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-12-19T18:46:41.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.292+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f21cc068f70 msgr2=0x7f21cc06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:41.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.292+0000 7f21e3230700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f21cc068f70 0x7f21cc06b420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f21c800b0d0 tx=0x7f21c8009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:41.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.292+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0faed0 msgr2=0x7f21dc0fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:41.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.292+0000 7f21e3230700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0faed0 0x7f21dc0fd9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f21d000eb70 tx=0x7f21d000b040 comp rx=0 tx=0).stop 2023-12-19T18:46:41.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.293+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 shutdown_connections 2023-12-19T18:46:41.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.293+0000 7f21e3230700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f21cc068f70 0x7f21cc06b420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.293+0000 7f21e3230700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f21dc0faed0 0x7f21dc0fd9f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.293+0000 7f21e3230700 1 --2- 172.21.15.59:0/4057031556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21dc0fdf60 0x7f21dc0fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.293+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 >> 172.21.15.59:0/4057031556 conn(0x7f21dc0f6950 msgr2=0x7f21dc101190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.293+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 shutdown_connections 2023-12-19T18:46:41.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.293+0000 7f21e3230700 1 -- 172.21.15.59:0/4057031556 wait complete. 2023-12-19T18:46:41.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.415+0000 7f7f512eb700 1 -- 172.21.15.59:0/864426732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0fcbf0 msgr2=0x7f7f4c0fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:41.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.415+0000 7f7f512eb700 1 --2- 172.21.15.59:0/864426732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0fcbf0 0x7f7f4c0fd000 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c004880 tx=0x7f7f3c009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:41.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.415+0000 7f7f512eb700 1 -- 172.21.15.59:0/864426732 shutdown_connections 2023-12-19T18:46:41.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.415+0000 7f7f512eb700 1 --2- 172.21.15.59:0/864426732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4c0fde00 0x7f7f4c0fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.415+0000 7f7f512eb700 1 --2- 172.21.15.59:0/864426732 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0fcbf0 0x7f7f4c0fd000 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.415+0000 7f7f512eb700 1 -- 172.21.15.59:0/864426732 >> 172.21.15.59:0/864426732 conn(0x7f7f4c0f82d0 msgr2=0x7f7f4c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:41.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.415+0000 7f7f512eb700 1 -- 172.21.15.59:0/864426732 shutdown_connections 2023-12-19T18:46:41.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.415+0000 7f7f512eb700 1 -- 172.21.15.59:0/864426732 wait complete. 2023-12-19T18:46:41.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f512eb700 1 Processor -- start 2023-12-19T18:46:41.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f512eb700 1 -- start start 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f512eb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4c0fcbf0 0x7f7f4c10d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f512eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0fde00 0x7f7f4c10da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f512eb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c10e010 con 0x7f7f4c0fde00 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f512eb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4c10e8b0 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f4a7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0fde00 0x7f7f4c10da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f4affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4c0fcbf0 0x7f7f4c10d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f4affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4c0fcbf0 0x7f7f4c10d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.59:33050/0 (socket says 172.21.15.59:33050) 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f4a7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0fde00 0x7f7f4c10da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:49348/0 (socket says 172.21.15.59:49348) 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.416+0000 7f7f4affd700 1 -- 172.21.15.59:0/1698243863 learned_addr learned my addr 172.21.15.59:0/1698243863 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.417+0000 7f7f4affd700 1 -- 172.21.15.59:0/1698243863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0fde00 msgr2=0x7f7f4c10da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.417+0000 7f7f4affd700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0fde00 0x7f7f4c10da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.417+0000 7f7f4affd700 1 -- 172.21.15.59:0/1698243863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f40005040 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.417+0000 7f7f4affd700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4c0fcbf0 0x7f7f4c10d4d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c000c00 tx=0x7f7f3c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:41.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.417+0000 7f7f337fe700 1 -- 172.21.15.59:0/1698243863 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f3c00e030 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.417+0000 7f7f337fe700 1 -- 172.21.15.59:0/1698243863 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f3c01c400 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.417+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c005040 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.417+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4c10eee0 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.417+0000 7f7f337fe700 1 -- 172.21.15.59:0/1698243863 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f3c012600 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.418+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f4c04bb80 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.419+0000 7f7f337fe700 1 -- 172.21.15.59:0/1698243863 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7f3c013a20 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.419+0000 7f7f337fe700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7f34068f60 0x7f7f3406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:46:41.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.419+0000 7f7f337fe700 1 -- 172.21.15.59:0/1698243863 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7f3c0887e0 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.419+0000 7f7f4a7fc700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7f34068f60 0x7f7f3406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:46:41.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.420+0000 7f7f4a7fc700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7f34068f60 0x7f7f3406b410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009fd0 tx=0x7f7f4000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:46:41.424 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.423+0000 7f7f337fe700 1 -- 172.21.15.59:0/1698243863 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7f3c0586f0 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.752+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7f4c10f270 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.752+0000 7f7f337fe700 1 -- 172.21.15.59:0/1698243863 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f7f3c058280 con 0x7f7f4c0fcbf0 2023-12-19T18:46:41.754 INFO:teuthology.orchestra.run.smithi059.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2023-12-19T18:46:41.754 INFO:teuthology.orchestra.run.smithi059.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2023-12-19T18:46:41.754 INFO:teuthology.orchestra.run.smithi059.stdout: fs cephfs has deprecated feature inline_data enabled. 2023-12-19T18:46:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7f34068f60 msgr2=0x7f7f3406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7f34068f60 0x7f7f3406b410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009fd0 tx=0x7f7f4000b040 comp rx=0 tx=0).stop 2023-12-19T18:46:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4c0fcbf0 msgr2=0x7f7f4c10d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:46:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4c0fcbf0 0x7f7f4c10d4d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c000c00 tx=0x7f7f3c009e30 comp rx=0 tx=0).stop 2023-12-19T18:46:41.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 shutdown_connections 2023-12-19T18:46:41.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f4c0fcbf0 0x7f7f4c10d4d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7f34068f60 0x7f7f3406b410 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 --2- 172.21.15.59:0/1698243863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f4c0fde00 0x7f7f4c10da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:46:41.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 >> 172.21.15.59:0/1698243863 conn(0x7f7f4c0f82d0 msgr2=0x7f7f4c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:46:41.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 shutdown_connections 2023-12-19T18:46:41.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:46:41.755+0000 7f7f512eb700 1 -- 172.21.15.59:0/1698243863 wait complete. 2023-12-19T18:46:42.198 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:42 smithi059 ceph-mon[80661]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:42.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:42 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1698243863' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-19T18:46:42.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:42 smithi089 ceph-mon[88459]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:46:42.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:42 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1698243863' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-19T18:46:43.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:43 smithi059 ceph-mon[80661]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:43.504 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:43 smithi089 ceph-mon[88459]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:45.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:45 smithi059 ceph-mon[80661]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:46.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:45 smithi089 ceph-mon[88459]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:47.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:47 smithi089 ceph-mon[88459]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:47.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:47 smithi059 ceph-mon[80661]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:49.611 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:49 smithi059 ceph-mon[80661]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:49.754 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:49 smithi089 ceph-mon[88459]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:51.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:51 smithi059 ceph-mon[80661]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:52.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:51 smithi089 ceph-mon[88459]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:53.178 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:52 smithi059 ceph-mon[80661]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:53.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:52 smithi089 ceph-mon[88459]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:54.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:53 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:46:54.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:53 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:46:55.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:54 smithi089 ceph-mon[88459]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:55.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:54 smithi059 ceph-mon[80661]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:57.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:57 smithi089 ceph-mon[88459]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:57.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:57 smithi059 ceph-mon[80661]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T18:46:59.610 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:46:59 smithi059 ceph-mon[80661]: pgmap v210: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:47:00.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:46:59 smithi089 ceph-mon[88459]: pgmap v210: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:47:01.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:01 smithi059 ceph-mon[80661]: pgmap v211: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:47:02.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:01 smithi089 ceph-mon[88459]: pgmap v211: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:47:03.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:02 smithi089 ceph-mon[88459]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:47:03.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:02 smithi059 ceph-mon[80661]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2023-12-19T18:47:05.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:05 smithi059 ceph-mon[80661]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T18:47:06.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:05 smithi089 ceph-mon[88459]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T18:47:07.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:07 smithi089 ceph-mon[88459]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T18:47:07.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:07.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:47:07.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:07 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:07.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:07 smithi059 ceph-mon[80661]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T18:47:07.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:07.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:47:07.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:07 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:08.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:08 smithi059 ceph-mon[80661]: Upgrade: Target is version 19.0.0-202-gc7586e95 (unknown) 2023-12-19T18:47:08.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:08 smithi059 ceph-mon[80661]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5'] 2023-12-19T18:47:08.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:08 smithi059 ceph-mon[80661]: Upgrade: Need to upgrade myself (mgr.smithi059.ccbnob) 2023-12-19T18:47:08.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:08 smithi059 ceph-mon[80661]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 on smithi089 2023-12-19T18:47:08.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:08 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:47:09.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:08 smithi089 ceph-mon[88459]: Upgrade: Target is version 19.0.0-202-gc7586e95 (unknown) 2023-12-19T18:47:09.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:08 smithi089 ceph-mon[88459]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5'] 2023-12-19T18:47:09.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:08 smithi089 ceph-mon[88459]: Upgrade: Need to upgrade myself (mgr.smithi059.ccbnob) 2023-12-19T18:47:09.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:08 smithi089 ceph-mon[88459]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 on smithi089 2023-12-19T18:47:09.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:08 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:47:09.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:09 smithi059 ceph-mon[80661]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T18:47:10.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:09 smithi089 ceph-mon[88459]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T18:47:11.915 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:11 smithi059 ceph-mon[80661]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T18:47:11.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.913+0000 7f9a59525700 1 -- 172.21.15.59:0/2598995246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fcbf0 msgr2=0x7f9a540fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:11.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.913+0000 7f9a59525700 1 --2- 172.21.15.59:0/2598995246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fcbf0 0x7f9a540fd000 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9a44004940 tx=0x7f9a44009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:11.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.913+0000 7f9a59525700 1 -- 172.21.15.59:0/2598995246 shutdown_connections 2023-12-19T18:47:11.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.913+0000 7f9a59525700 1 --2- 172.21.15.59:0/2598995246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a540fde00 0x7f9a540fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:11.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.913+0000 7f9a59525700 1 --2- 172.21.15.59:0/2598995246 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fcbf0 0x7f9a540fd000 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:11.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.913+0000 7f9a59525700 1 -- 172.21.15.59:0/2598995246 >> 172.21.15.59:0/2598995246 conn(0x7f9a540f82d0 msgr2=0x7f9a540fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:11.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.913+0000 7f9a59525700 1 -- 172.21.15.59:0/2598995246 shutdown_connections 2023-12-19T18:47:11.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.913+0000 7f9a59525700 1 -- 172.21.15.59:0/2598995246 wait complete. 2023-12-19T18:47:11.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a59525700 1 Processor -- start 2023-12-19T18:47:11.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a59525700 1 -- start start 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a59525700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a540fcbf0 0x7f9a54071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a59525700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fde00 0x7f9a5406ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a59525700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a54070590 con 0x7f9a540fde00 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a59525700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a54070700 con 0x7f9a540fcbf0 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a527fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fde00 0x7f9a5406ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a527fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fde00 0x7f9a5406ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58434/0 (socket says 172.21.15.59:58434) 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a527fc700 1 -- 172.21.15.59:0/1414579975 learned_addr learned my addr 172.21.15.59:0/1414579975 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.914+0000 7f9a52ffd700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a540fcbf0 0x7f9a54071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.915+0000 7f9a52ffd700 1 -- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fde00 msgr2=0x7f9a5406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:11.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.915+0000 7f9a52ffd700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fde00 0x7f9a5406ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.915+0000 7f9a52ffd700 1 -- 172.21.15.59:0/1414579975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a44005040 con 0x7f9a540fcbf0 2023-12-19T18:47:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.915+0000 7f9a527fc700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fde00 0x7f9a5406ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:47:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.915+0000 7f9a52ffd700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a540fcbf0 0x7f9a54071b10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400b0d0 tx=0x7f9a44009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.915+0000 7f9a3b7fe700 1 -- 172.21.15.59:0/1414579975 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a4400e070 con 0x7f9a540fcbf0 2023-12-19T18:47:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.915+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a54070980 con 0x7f9a540fcbf0 2023-12-19T18:47:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.915+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a54115500 con 0x7f9a540fcbf0 2023-12-19T18:47:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.915+0000 7f9a3b7fe700 1 -- 172.21.15.59:0/1414579975 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a4400b410 con 0x7f9a540fcbf0 2023-12-19T18:47:11.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.916+0000 7f9a3b7fe700 1 -- 172.21.15.59:0/1414579975 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a44012600 con 0x7f9a540fcbf0 2023-12-19T18:47:11.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.917+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5404bb80 con 0x7f9a540fcbf0 2023-12-19T18:47:11.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.917+0000 7f9a3b7fe700 1 -- 172.21.15.59:0/1414579975 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9a4400b9f0 con 0x7f9a540fcbf0 2023-12-19T18:47:11.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.918+0000 7f9a3b7fe700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9a3c068f60 0x7f9a3c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:11.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.918+0000 7f9a3b7fe700 1 -- 172.21.15.59:0/1414579975 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9a440876b0 con 0x7f9a540fcbf0 2023-12-19T18:47:11.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.918+0000 7f9a527fc700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9a3c068f60 0x7f9a3c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:11.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.918+0000 7f9a527fc700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9a3c068f60 0x7f9a3c06b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a480041d0 tx=0x7f9a4800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:11.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:11.926+0000 7f9a3b7fe700 1 -- 172.21.15.59:0/1414579975 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9a44057660 con 0x7f9a540fcbf0 2023-12-19T18:47:12.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:11 smithi089 ceph-mon[88459]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T18:47:12.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.146+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a54102750 con 0x7f9a3c068f60 2023-12-19T18:47:12.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.147+0000 7f9a3b7fe700 1 -- 172.21.15.59:0/1414579975 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f9a54102750 con 0x7f9a3c068f60 2023-12-19T18:47:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9a3c068f60 msgr2=0x7f9a3c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9a3c068f60 0x7f9a3c06b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a480041d0 tx=0x7f9a4800b040 comp rx=0 tx=0).stop 2023-12-19T18:47:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a540fcbf0 msgr2=0x7f9a54071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a540fcbf0 0x7f9a54071b10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400b0d0 tx=0x7f9a44009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 shutdown_connections 2023-12-19T18:47:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a540fcbf0 0x7f9a54071b10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9a3c068f60 0x7f9a3c06b410 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 --2- 172.21.15.59:0/1414579975 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9a540fde00 0x7f9a5406ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 >> 172.21.15.59:0/1414579975 conn(0x7f9a540f82d0 msgr2=0x7f9a54101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:12.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 shutdown_connections 2023-12-19T18:47:12.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.149+0000 7f9a59525700 1 -- 172.21.15.59:0/1414579975 wait complete. 2023-12-19T18:47:12.161 INFO:teuthology.orchestra.run.smithi059.stdout:true 2023-12-19T18:47:12.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.259+0000 7f6a96746700 1 -- 172.21.15.59:0/258856504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a901020e0 msgr2=0x7f6a901044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:12.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.259+0000 7f6a96746700 1 --2- 172.21.15.59:0/258856504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a901020e0 0x7f6a901044c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6a84004940 tx=0x7f6a84009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:12.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.259+0000 7f6a96746700 1 -- 172.21.15.59:0/258856504 shutdown_connections 2023-12-19T18:47:12.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.259+0000 7f6a96746700 1 --2- 172.21.15.59:0/258856504 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a901020e0 0x7f6a901044c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.259+0000 7f6a96746700 1 --2- 172.21.15.59:0/258856504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a90066090 0x7f6a90101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.259+0000 7f6a96746700 1 -- 172.21.15.59:0/258856504 >> 172.21.15.59:0/258856504 conn(0x7f6a900f7190 msgr2=0x7f6a900f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:12.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.259+0000 7f6a96746700 1 -- 172.21.15.59:0/258856504 shutdown_connections 2023-12-19T18:47:12.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.259+0000 7f6a96746700 1 -- 172.21.15.59:0/258856504 wait complete. 2023-12-19T18:47:12.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.260+0000 7f6a96746700 1 Processor -- start 2023-12-19T18:47:12.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.260+0000 7f6a96746700 1 -- start start 2023-12-19T18:47:12.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.260+0000 7f6a96746700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a90066090 0x7f6a9010f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:12.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.260+0000 7f6a96746700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a901020e0 0x7f6a9010fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:12.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a8f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a901020e0 0x7f6a9010fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:12.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a8f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a901020e0 0x7f6a9010fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58450/0 (socket says 172.21.15.59:58450) 2023-12-19T18:47:12.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a8f7fe700 1 -- 172.21.15.59:0/2902527276 learned_addr learned my addr 172.21.15.59:0/2902527276 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:12.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a90110390 con 0x7f6a901020e0 2023-12-19T18:47:12.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a90110c30 con 0x7f6a90066090 2023-12-19T18:47:12.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a8ffff700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a90066090 0x7f6a9010f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:12.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a8ffff700 1 -- 172.21.15.59:0/2902527276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a901020e0 msgr2=0x7f6a9010fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:12.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a8ffff700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a901020e0 0x7f6a9010fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a8ffff700 1 -- 172.21.15.59:0/2902527276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a84005040 con 0x7f6a90066090 2023-12-19T18:47:12.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.261+0000 7f6a8ffff700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a90066090 0x7f6a9010f850 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009fd0 tx=0x7f6a8000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:12.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.262+0000 7f6a8cff9700 1 -- 172.21.15.59:0/2902527276 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a80004e90 con 0x7f6a90066090 2023-12-19T18:47:12.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.262+0000 7f6a8cff9700 1 -- 172.21.15.59:0/2902527276 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a8000f040 con 0x7f6a90066090 2023-12-19T18:47:12.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.262+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a90110eb0 con 0x7f6a90066090 2023-12-19T18:47:12.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.262+0000 7f6a8cff9700 1 -- 172.21.15.59:0/2902527276 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a8000c9b0 con 0x7f6a90066090 2023-12-19T18:47:12.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.262+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a901113f0 con 0x7f6a90066090 2023-12-19T18:47:12.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.264+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a901096f0 con 0x7f6a90066090 2023-12-19T18:47:12.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.264+0000 7f6a8cff9700 1 -- 172.21.15.59:0/2902527276 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6a80007cc0 con 0x7f6a90066090 2023-12-19T18:47:12.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.265+0000 7f6a8cff9700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6a78068f60 0x7f6a7806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:12.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.265+0000 7f6a8cff9700 1 -- 172.21.15.59:0/2902527276 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6a800867f0 con 0x7f6a90066090 2023-12-19T18:47:12.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.265+0000 7f6a8f7fe700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6a78068f60 0x7f6a7806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:12.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.266+0000 7f6a8f7fe700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6a78068f60 0x7f6a7806b410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6a840135e0 tx=0x7f6a84009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:12.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.269+0000 7f6a8cff9700 1 -- 172.21.15.59:0/2902527276 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6a8004da00 con 0x7f6a90066090 2023-12-19T18:47:12.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.591+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a90111660 con 0x7f6a78068f60 2023-12-19T18:47:12.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.593+0000 7f6a8cff9700 1 -- 172.21.15.59:0/2902527276 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f6a90111660 con 0x7f6a78068f60 2023-12-19T18:47:12.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6a78068f60 msgr2=0x7f6a7806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:12.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6a78068f60 0x7f6a7806b410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6a840135e0 tx=0x7f6a84009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:12.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a90066090 msgr2=0x7f6a9010f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:12.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a90066090 0x7f6a9010f850 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009fd0 tx=0x7f6a8000b040 comp rx=0 tx=0).stop 2023-12-19T18:47:12.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 shutdown_connections 2023-12-19T18:47:12.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a90066090 0x7f6a9010f850 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6a78068f60 0x7f6a7806b410 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 --2- 172.21.15.59:0/2902527276 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6a901020e0 0x7f6a9010fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 >> 172.21.15.59:0/2902527276 conn(0x7f6a900f7190 msgr2=0x7f6a900f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:12.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 shutdown_connections 2023-12-19T18:47:12.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.596+0000 7f6a96746700 1 -- 172.21.15.59:0/2902527276 wait complete. 2023-12-19T18:47:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.748+0000 7f62af9a1700 1 -- 172.21.15.59:0/3350369610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80fcd00 msgr2=0x7f62a80fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.748+0000 7f62af9a1700 1 --2- 172.21.15.59:0/3350369610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80fcd00 0x7f62a80fd150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6298004940 tx=0x7f6298009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.749+0000 7f62af9a1700 1 -- 172.21.15.59:0/3350369610 shutdown_connections 2023-12-19T18:47:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.749+0000 7f62af9a1700 1 --2- 172.21.15.59:0/3350369610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80fcd00 0x7f62a80fd150 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.749+0000 7f62af9a1700 1 --2- 172.21.15.59:0/3350369610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a80fbaf0 0x7f62a80fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.749+0000 7f62af9a1700 1 -- 172.21.15.59:0/3350369610 >> 172.21.15.59:0/3350369610 conn(0x7f62a80f7190 msgr2=0x7f62a80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.749+0000 7f62af9a1700 1 -- 172.21.15.59:0/3350369610 shutdown_connections 2023-12-19T18:47:12.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.749+0000 7f62af9a1700 1 -- 172.21.15.59:0/3350369610 wait complete. 2023-12-19T18:47:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.749+0000 7f62af9a1700 1 Processor -- start 2023-12-19T18:47:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.750+0000 7f62af9a1700 1 -- start start 2023-12-19T18:47:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.750+0000 7f62af9a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80fbaf0 0x7f62a810c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.750+0000 7f62af9a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a80fcd00 0x7f62a810c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.750+0000 7f62af9a1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a810cf10 con 0x7f62a80fbaf0 2023-12-19T18:47:12.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.750+0000 7f62af9a1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a810d7b0 con 0x7f62a80fcd00 2023-12-19T18:47:12.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.751+0000 7f62acf3c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a80fcd00 0x7f62a810c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:12.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.751+0000 7f62ad73d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80fbaf0 0x7f62a810c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:12.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.751+0000 7f62acf3c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a80fcd00 0x7f62a810c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.59:50866/0 (socket says 172.21.15.59:50866) 2023-12-19T18:47:12.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.751+0000 7f62acf3c700 1 -- 172.21.15.59:0/124604115 learned_addr learned my addr 172.21.15.59:0/124604115 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:12.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.751+0000 7f62acf3c700 1 -- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80fbaf0 msgr2=0x7f62a810c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:12.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.751+0000 7f62acf3c700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80fbaf0 0x7f62a810c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:12.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.751+0000 7f62acf3c700 1 -- 172.21.15.59:0/124604115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6298005040 con 0x7f62a80fcd00 2023-12-19T18:47:12.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.751+0000 7f62ad73d700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80fbaf0 0x7f62a810c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:47:12.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.751+0000 7f62acf3c700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a80fcd00 0x7f62a810c910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6298000c00 tx=0x7f6298009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:12.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.752+0000 7f629dffb700 1 -- 172.21.15.59:0/124604115 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6298021070 con 0x7f62a80fcd00 2023-12-19T18:47:12.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.752+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a810da30 con 0x7f62a80fcd00 2023-12-19T18:47:12.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.752+0000 7f629dffb700 1 -- 172.21.15.59:0/124604115 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f629801c400 con 0x7f62a80fcd00 2023-12-19T18:47:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.752+0000 7f629dffb700 1 -- 172.21.15.59:0/124604115 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f629800e030 con 0x7f62a80fcd00 2023-12-19T18:47:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.752+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a810df70 con 0x7f62a80fcd00 2023-12-19T18:47:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.753+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a804bb80 con 0x7f62a80fcd00 2023-12-19T18:47:12.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.753+0000 7f629dffb700 1 -- 172.21.15.59:0/124604115 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f629801ca00 con 0x7f62a80fcd00 2023-12-19T18:47:12.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.754+0000 7f629dffb700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6294068f60 0x7f629406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:12.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.754+0000 7f629dffb700 1 -- 172.21.15.59:0/124604115 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6298087fb0 con 0x7f62a80fcd00 2023-12-19T18:47:12.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.754+0000 7f62ad73d700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6294068f60 0x7f629406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:12.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.755+0000 7f62ad73d700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6294068f60 0x7f629406b410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009b80 tx=0x7f62a400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:12.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.758+0000 7f629dffb700 1 -- 172.21.15.59:0/124604115 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6298057f60 con 0x7f62a80fcd00 2023-12-19T18:47:12.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:12.989+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f62a810e410 con 0x7f6294068f60 2023-12-19T18:47:13.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.000+0000 7f629dffb700 1 -- 172.21.15.59:0/124604115 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f62a810e410 con 0x7f6294068f60 2023-12-19T18:47:13.002 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-19T18:47:13.002 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (5m) 57s ago 6m 25.1M - 0.25.0 c8568f914cd2 528f56ce4fdd 2023-12-19T18:47:13.002 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (6m) 57s ago 6m 6660k - 17.2.7-131-ga88f3edf f9ecdbaa1030 03ae34e0a398 2023-12-19T18:47:13.002 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi089 smithi089 running (5m) 64s ago 5m 6665k - 17.2.7-131-ga88f3edf f9ecdbaa1030 8a39e517426d 2023-12-19T18:47:13.002 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (5m) 57s ago 6m 87.0M - 9.4.7 954c08fa6188 a19de18455de 2023-12-19T18:47:13.002 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.abqklt smithi059 running (83s) 57s ago 83s 10.4M - 17.2.7-131-ga88f3edf f9ecdbaa1030 c303f5a4640b 2023-12-19T18:47:13.003 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.rtqzmc smithi059 running (77s) 57s ago 76s 14.2M - 17.2.7-131-ga88f3edf f9ecdbaa1030 a2c3ca2a9ed2 2023-12-19T18:47:13.003 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi089.dfrkla smithi089 running (74s) 64s ago 74s 14.5M - 17.2.7-131-ga88f3edf f9ecdbaa1030 f707fc4d33e4 2023-12-19T18:47:13.003 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi089.dpgsbt smithi089 running (81s) 64s ago 80s 12.1M - 17.2.7-131-ga88f3edf f9ecdbaa1030 ce89c8fa8ca4 2023-12-19T18:47:13.003 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.ccbnob smithi059 *:9283 running (8m) 57s ago 8m 479M - 17.2.7-131-ga88f3edf f9ecdbaa1030 9ce46f56adc1 2023-12-19T18:47:13.003 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi089.isaaav smithi089 *:8443,9283 running (5m) 64s ago 5m 434M - 17.2.7-131-ga88f3edf f9ecdbaa1030 5890af67d417 2023-12-19T18:47:13.003 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (8m) 57s ago 8m 45.1M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 297c7051711f 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi089 smithi089 running (5m) 64s ago 5m 33.0M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 5ac855c460fa 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (6m) 57s ago 6m 16.0M - 1.5.0 0da6a335fe13 bfe5bff75fbd 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi089 smithi089 *:9100 running (5m) 64s ago 5m 15.7M - 1.5.0 0da6a335fe13 057c3879e7a2 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (4m) 57s ago 4m 48.2M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 87e75bce1c5b 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (4m) 57s ago 4m 48.5M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 1b133ef58db0 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (4m) 57s ago 4m 45.0M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 0f56936ea44c 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi089 running (3m) 64s ago 3m 44.7M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 3eafaf27a901 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi089 running (3m) 64s ago 3m 44.0M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 cc630810bb79 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi089 running (2m) 64s ago 2m 41.3M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 f1d513a4069b 2023-12-19T18:47:13.004 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (5m) 57s ago 6m 52.1M - 2.43.0 a07b618ecd1d 5884a00b2f95 2023-12-19T18:47:13.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.003+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6294068f60 msgr2=0x7f629406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.003+0000 7f62af9a1700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6294068f60 0x7f629406b410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009b80 tx=0x7f62a400b040 comp rx=0 tx=0).stop 2023-12-19T18:47:13.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.003+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a80fcd00 msgr2=0x7f62a810c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.003+0000 7f62af9a1700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a80fcd00 0x7f62a810c910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6298000c00 tx=0x7f6298009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:13.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.003+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 shutdown_connections 2023-12-19T18:47:13.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.004+0000 7f62af9a1700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f6294068f60 0x7f629406b410 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.004+0000 7f62af9a1700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62a80fbaf0 0x7f62a810c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.004+0000 7f62af9a1700 1 --2- 172.21.15.59:0/124604115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62a80fcd00 0x7f62a810c910 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.004+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 >> 172.21.15.59:0/124604115 conn(0x7f62a80f7190 msgr2=0x7f62a80fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:13.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.004+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 shutdown_connections 2023-12-19T18:47:13.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.004+0000 7f62af9a1700 1 -- 172.21.15.59:0/124604115 wait complete. 2023-12-19T18:47:13.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.110+0000 7f985695b700 1 -- 172.21.15.59:0/1516252123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 msgr2=0x7f98500fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.110+0000 7f985695b700 1 --2- 172.21.15.59:0/1516252123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 0x7f98500fd150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9844004940 tx=0x7f9844009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:13.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.110+0000 7f985695b700 1 -- 172.21.15.59:0/1516252123 shutdown_connections 2023-12-19T18:47:13.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.110+0000 7f985695b700 1 --2- 172.21.15.59:0/1516252123 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 0x7f98500fd150 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.110+0000 7f985695b700 1 --2- 172.21.15.59:0/1516252123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500fbaf0 0x7f98500fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.110+0000 7f985695b700 1 -- 172.21.15.59:0/1516252123 >> 172.21.15.59:0/1516252123 conn(0x7f98500f71b0 msgr2=0x7f98500f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:13.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.110+0000 7f985695b700 1 -- 172.21.15.59:0/1516252123 shutdown_connections 2023-12-19T18:47:13.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.110+0000 7f985695b700 1 -- 172.21.15.59:0/1516252123 wait complete. 2023-12-19T18:47:13.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f985695b700 1 Processor -- start 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f985695b700 1 -- start start 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f985695b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500fbaf0 0x7f985010d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f985695b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 0x7f985010d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f984ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500fbaf0 0x7f985010d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f984ffff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500fbaf0 0x7f985010d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.59:50876/0 (socket says 172.21.15.59:50876) 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f984f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 0x7f985010d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f984f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 0x7f985010d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58488/0 (socket says 172.21.15.59:58488) 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f985695b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985010dfb0 con 0x7f98500fcd00 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f985695b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985010e850 con 0x7f98500fbaf0 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.111+0000 7f984ffff700 1 -- 172.21.15.59:0/2898767149 learned_addr learned my addr 172.21.15.59:0/2898767149 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.112+0000 7f984ffff700 1 -- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 msgr2=0x7f985010d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.112+0000 7f984ffff700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 0x7f985010d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.112+0000 7f984ffff700 1 -- 172.21.15.59:0/2898767149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9844005040 con 0x7f98500fbaf0 2023-12-19T18:47:13.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.112+0000 7f984f7fe700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 0x7f985010d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:47:13.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.112+0000 7f984ffff700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500fbaf0 0x7f985010d470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9840010e80 tx=0x7f984000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:13.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.112+0000 7f984cff9700 1 -- 172.21.15.59:0/2898767149 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9840015070 con 0x7f98500fbaf0 2023-12-19T18:47:13.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.112+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985010ead0 con 0x7f98500fbaf0 2023-12-19T18:47:13.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.113+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985010f010 con 0x7f98500fbaf0 2023-12-19T18:47:13.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.113+0000 7f984cff9700 1 -- 172.21.15.59:0/2898767149 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f984000bde0 con 0x7f98500fbaf0 2023-12-19T18:47:13.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.113+0000 7f984cff9700 1 -- 172.21.15.59:0/2898767149 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f984001b470 con 0x7f98500fbaf0 2023-12-19T18:47:13.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.114+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983c0052e0 con 0x7f98500fbaf0 2023-12-19T18:47:13.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.115+0000 7f984cff9700 1 -- 172.21.15.59:0/2898767149 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f984000b930 con 0x7f98500fbaf0 2023-12-19T18:47:13.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.115+0000 7f984cff9700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9838068f70 0x7f983806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:13.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.115+0000 7f984f7fe700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9838068f70 0x7f983806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:13.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.116+0000 7f984cff9700 1 -- 172.21.15.59:0/2898767149 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f984008aff0 con 0x7f98500fbaf0 2023-12-19T18:47:13.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.116+0000 7f984f7fe700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9838068f70 0x7f983806b420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9844013600 tx=0x7f9844009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:13.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.119+0000 7f984cff9700 1 -- 172.21.15.59:0/2898767149 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f984005aec0 con 0x7f98500fbaf0 2023-12-19T18:47:13.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:12 smithi089 ceph-mon[88459]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:13.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:12 smithi089 ceph-mon[88459]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T18:47:13.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:12 smithi089 ceph-mon[88459]: from='client.24393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:13.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.402+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f983c005d00 con 0x7f98500fbaf0 2023-12-19T18:47:13.403 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:12 smithi059 ceph-mon[80661]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:13.403 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:12 smithi059 ceph-mon[80661]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T18:47:13.403 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:12 smithi059 ceph-mon[80661]: from='client.24393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:13.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.402+0000 7f984cff9700 1 -- 172.21.15.59:0/2898767149 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f984005aa50 con 0x7f98500fbaf0 2023-12-19T18:47:13.404 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-12-19T18:47:13.404 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": { 2023-12-19T18:47:13.404 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 2 2023-12-19T18:47:13.404 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": { 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 2 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": { 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 6 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": { 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 4 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: "overall": { 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 14 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout: } 2023-12-19T18:47:13.405 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-12-19T18:47:13.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9838068f70 msgr2=0x7f983806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9838068f70 0x7f983806b420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9844013600 tx=0x7f9844009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:13.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500fbaf0 msgr2=0x7f985010d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500fbaf0 0x7f985010d470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9840010e80 tx=0x7f984000b040 comp rx=0 tx=0).stop 2023-12-19T18:47:13.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 shutdown_connections 2023-12-19T18:47:13.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98500fbaf0 0x7f985010d470 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9838068f70 0x7f983806b420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 --2- 172.21.15.59:0/2898767149 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f98500fcd00 0x7f985010d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 >> 172.21.15.59:0/2898767149 conn(0x7f98500f71b0 msgr2=0x7f98500fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:13.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 shutdown_connections 2023-12-19T18:47:13.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.405+0000 7f985695b700 1 -- 172.21.15.59:0/2898767149 wait complete. 2023-12-19T18:47:13.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.518+0000 7fd2717de700 1 -- 172.21.15.59:0/2384451930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd26c0fd730 msgr2=0x7fd26c0ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.518+0000 7fd2717de700 1 --2- 172.21.15.59:0/2384451930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd26c0fd730 0x7fd26c0ffb10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd25c004940 tx=0x7fd25c009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:13.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.518+0000 7fd2717de700 1 -- 172.21.15.59:0/2384451930 shutdown_connections 2023-12-19T18:47:13.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.518+0000 7fd2717de700 1 --2- 172.21.15.59:0/2384451930 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c100050 0x7fd26c102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.518+0000 7fd2717de700 1 --2- 172.21.15.59:0/2384451930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd26c0fd730 0x7fd26c0ffb10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.518+0000 7fd2717de700 1 -- 172.21.15.59:0/2384451930 >> 172.21.15.59:0/2384451930 conn(0x7fd26c0f71d0 msgr2=0x7fd26c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:13.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.518+0000 7fd2717de700 1 -- 172.21.15.59:0/2384451930 shutdown_connections 2023-12-19T18:47:13.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.518+0000 7fd2717de700 1 -- 172.21.15.59:0/2384451930 wait complete. 2023-12-19T18:47:13.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd2717de700 1 Processor -- start 2023-12-19T18:47:13.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd2717de700 1 -- start start 2023-12-19T18:47:13.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd2717de700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c0fd730 0x7fd26c10d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:13.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd2717de700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd26c100050 0x7fd26c10d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:13.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd2717de700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c10dfa0 con 0x7fd26c100050 2023-12-19T18:47:13.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd2717de700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26c10e840 con 0x7fd26c0fd730 2023-12-19T18:47:13.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd26affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c0fd730 0x7fd26c10d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:13.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd26affd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c0fd730 0x7fd26c10d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.59:50888/0 (socket says 172.21.15.59:50888) 2023-12-19T18:47:13.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd26affd700 1 -- 172.21.15.59:0/344762899 learned_addr learned my addr 172.21.15.59:0/344762899 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:13.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.519+0000 7fd26a7fc700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd26c100050 0x7fd26c10d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.520+0000 7fd26affd700 1 -- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd26c100050 msgr2=0x7fd26c10d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.520+0000 7fd26affd700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd26c100050 0x7fd26c10d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.520+0000 7fd26affd700 1 -- 172.21.15.59:0/344762899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c005040 con 0x7fd26c0fd730 2023-12-19T18:47:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.520+0000 7fd26a7fc700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd26c100050 0x7fd26c10d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:47:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.520+0000 7fd26affd700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c0fd730 0x7fd26c10d460 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009fd0 tx=0x7fd25c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.520+0000 7fd2537fe700 1 -- 172.21.15.59:0/344762899 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd25c021070 con 0x7fd26c0fd730 2023-12-19T18:47:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.520+0000 7fd2537fe700 1 -- 172.21.15.59:0/344762899 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd25c01c400 con 0x7fd26c0fd730 2023-12-19T18:47:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.520+0000 7fd2537fe700 1 -- 172.21.15.59:0/344762899 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd25c00e070 con 0x7fd26c0fd730 2023-12-19T18:47:13.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.520+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26c10eac0 con 0x7fd26c0fd730 2023-12-19T18:47:13.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.521+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26c10ef50 con 0x7fd26c0fd730 2023-12-19T18:47:13.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.522+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c04bb80 con 0x7fd26c0fd730 2023-12-19T18:47:13.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.523+0000 7fd2537fe700 1 -- 172.21.15.59:0/344762899 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd25c01ca00 con 0x7fd26c0fd730 2023-12-19T18:47:13.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.523+0000 7fd2537fe700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd254068f60 0x7fd25406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:13.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.523+0000 7fd2537fe700 1 -- 172.21.15.59:0/344762899 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd25c088080 con 0x7fd26c0fd730 2023-12-19T18:47:13.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.523+0000 7fd26a7fc700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd254068f60 0x7fd25406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:13.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.524+0000 7fd26a7fc700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd254068f60 0x7fd25406b410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd260009b80 tx=0x7fd26000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:13.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.527+0000 7fd2537fe700 1 -- 172.21.15.59:0/344762899 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd25c057ef0 con 0x7fd26c0fd730 2023-12-19T18:47:13.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.787+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd26c10f380 con 0x7fd26c0fd730 2023-12-19T18:47:13.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.788+0000 7fd2537fe700 1 -- 172.21.15.59:0/344762899 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1805 (secure 0 0 0) 0x7fd25c057a80 con 0x7fd26c0fd730 2023-12-19T18:47:13.789 INFO:teuthology.orchestra.run.smithi059.stdout:e15 2023-12-19T18:47:13.789 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-19T18:47:13.789 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:47:13.789 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2023-12-19T18:47:13.789 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:47:13.789 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2023-12-19T18:47:13.789 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2023-12-19T18:47:13.789 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 15 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:created 2023-12-19T18:45:45.631056+0000 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2023-12-19T18:46:03.615908+0000 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 47 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:in 0 2023-12-19T18:47:13.790 INFO:teuthology.orchestra.run.smithi059.stdout:up {0=14474} 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data enabled 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 1 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.rtqzmc{0:14474} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:Standby daemons: 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.abqklt{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.59:6826/1872103080,v1:172.21.15.59:6827/1872103080] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi089.dpgsbt{-1:24291} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.89:6824/1622105230,v1:172.21.15.89:6825/1622105230] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:47:13.791 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi089.dfrkla{-1:24301} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.89:6826/3691364030,v1:172.21.15.89:6827/3691364030] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:47:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd254068f60 msgr2=0x7fd25406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd254068f60 0x7fd25406b410 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd260009b80 tx=0x7fd26000b040 comp rx=0 tx=0).stop 2023-12-19T18:47:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c0fd730 msgr2=0x7fd26c10d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c0fd730 0x7fd26c10d460 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009fd0 tx=0x7fd25c009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:13.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 shutdown_connections 2023-12-19T18:47:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd26c0fd730 0x7fd26c10d460 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009fd0 tx=0x7fd25c009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fd254068f60 0x7fd25406b410 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 --2- 172.21.15.59:0/344762899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd26c100050 0x7fd26c10d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 >> 172.21.15.59:0/344762899 conn(0x7fd26c0f71d0 msgr2=0x7fd26c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.790+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 shutdown_connections 2023-12-19T18:47:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.791+0000 7fd2717de700 1 -- 172.21.15.59:0/344762899 wait complete. 2023-12-19T18:47:13.793 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 15 2023-12-19T18:47:13.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.957+0000 7f640e983700 1 -- 172.21.15.59:0/1543609717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 msgr2=0x7f64080fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.957+0000 7f640e983700 1 --2- 172.21.15.59:0/1543609717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 0x7f64080fbf00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f63f8004940 tx=0x7f63f8009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:13.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.957+0000 7f640e983700 1 -- 172.21.15.59:0/1543609717 shutdown_connections 2023-12-19T18:47:13.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.957+0000 7f640e983700 1 --2- 172.21.15.59:0/1543609717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fcd00 0x7f64080fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.957+0000 7f640e983700 1 --2- 172.21.15.59:0/1543609717 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 0x7f64080fbf00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.957+0000 7f640e983700 1 -- 172.21.15.59:0/1543609717 >> 172.21.15.59:0/1543609717 conn(0x7f64080f7190 msgr2=0x7f64080f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:13.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.958+0000 7f640e983700 1 -- 172.21.15.59:0/1543609717 shutdown_connections 2023-12-19T18:47:13.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.958+0000 7f640e983700 1 -- 172.21.15.59:0/1543609717 wait complete. 2023-12-19T18:47:13.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.959+0000 7f640e983700 1 Processor -- start 2023-12-19T18:47:13.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.959+0000 7f640e983700 1 -- start start 2023-12-19T18:47:13.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.960+0000 7f640e983700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 0x7f640810b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:13.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.960+0000 7f640e983700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fcd00 0x7f640810b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:13.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.960+0000 7f640e983700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640810be80 con 0x7f64080fbaf0 2023-12-19T18:47:13.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.960+0000 7f6407fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 0x7f640810b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:13.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.960+0000 7f6407fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 0x7f640810b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58534/0 (socket says 172.21.15.59:58534) 2023-12-19T18:47:13.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.960+0000 7f6407fff700 1 -- 172.21.15.59:0/2771012170 learned_addr learned my addr 172.21.15.59:0/2771012170 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:13.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.960+0000 7f640e983700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640810c720 con 0x7f64080fcd00 2023-12-19T18:47:13.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.961+0000 7f64077fe700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fcd00 0x7f640810b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:13.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.961+0000 7f6407fff700 1 -- 172.21.15.59:0/2771012170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fcd00 msgr2=0x7f640810b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:13.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.961+0000 7f6407fff700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fcd00 0x7f640810b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:13.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.961+0000 7f6407fff700 1 -- 172.21.15.59:0/2771012170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc005040 con 0x7f64080fbaf0 2023-12-19T18:47:13.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.961+0000 7f64077fe700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fcd00 0x7f640810b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T18:47:13.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.961+0000 7f6407fff700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 0x7f640810b340 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009fd0 tx=0x7f63f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:13.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.966+0000 7f6404ff9700 1 -- 172.21.15.59:0/2771012170 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63f8021070 con 0x7f64080fbaf0 2023-12-19T18:47:13.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.966+0000 7f6404ff9700 1 -- 172.21.15.59:0/2771012170 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63f801c400 con 0x7f64080fbaf0 2023-12-19T18:47:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.966+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f8005040 con 0x7f64080fbaf0 2023-12-19T18:47:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.966+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640810cc30 con 0x7f64080fbaf0 2023-12-19T18:47:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.966+0000 7f6404ff9700 1 -- 172.21.15.59:0/2771012170 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63f800e070 con 0x7f64080fbaf0 2023-12-19T18:47:13.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.967+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640804bb80 con 0x7f64080fbaf0 2023-12-19T18:47:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.968+0000 7f6404ff9700 1 -- 172.21.15.59:0/2771012170 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f63f800b650 con 0x7f64080fbaf0 2023-12-19T18:47:13.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.969+0000 7f6404ff9700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f63f0069020 0x7f63f006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:13.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.969+0000 7f6404ff9700 1 -- 172.21.15.59:0/2771012170 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f63f8087f30 con 0x7f64080fbaf0 2023-12-19T18:47:13.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.969+0000 7f64077fe700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f63f0069020 0x7f63f006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:13.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.969+0000 7f64077fe700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f63f0069020 0x7f63f006b4d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f63fc004b40 tx=0x7f63fc0048d0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:13.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:13.977+0000 7f6404ff9700 1 -- 172.21.15.59:0/2771012170 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f63f8057ee0 con 0x7f64080fbaf0 2023-12-19T18:47:14.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:13 smithi059 ceph-mon[80661]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:14.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:13 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/2898767149' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-19T18:47:14.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:13 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/344762899' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:47:14.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:13 smithi089 ceph-mon[88459]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:14.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:13 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/2898767149' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-19T18:47:14.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:13 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/344762899' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:47:14.348 INFO:tasks.workunit.client.1.smithi089.stderr:Updating files: 96% (11200/11620) Updating files: 97% (11272/11620) Updating files: 98% (11388/11620) Updating files: 99% (11504/11620) Updating files: 100% (11620/11620) Updating files: 100% (11620/11620), done. 2023-12-19T18:47:14.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.370+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f640810d440 con 0x7f63f0069020 2023-12-19T18:47:14.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.371+0000 7f6404ff9700 1 -- 172.21.15.59:0/2771012170 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f640810d440 con 0x7f63f0069020 2023-12-19T18:47:14.372 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-12-19T18:47:14.372 INFO:teuthology.orchestra.run.smithi059.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5", 2023-12-19T18:47:14.372 INFO:teuthology.orchestra.run.smithi059.stdout: "in_progress": true, 2023-12-19T18:47:14.372 INFO:teuthology.orchestra.run.smithi059.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-19T18:47:14.372 INFO:teuthology.orchestra.run.smithi059.stdout: "services_complete": [], 2023-12-19T18:47:14.373 INFO:teuthology.orchestra.run.smithi059.stdout: "progress": "0/21 daemons upgraded", 2023-12-19T18:47:14.373 INFO:teuthology.orchestra.run.smithi059.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 image on host smithi089", 2023-12-19T18:47:14.373 INFO:teuthology.orchestra.run.smithi059.stdout: "is_paused": false 2023-12-19T18:47:14.373 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-12-19T18:47:14.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f63f0069020 msgr2=0x7f63f006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:14.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f63f0069020 0x7f63f006b4d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f63fc004b40 tx=0x7f63fc0048d0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 msgr2=0x7f640810b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:14.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 0x7f640810b340 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009fd0 tx=0x7f63f8009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:14.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 shutdown_connections 2023-12-19T18:47:14.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f63f0069020 0x7f63f006b4d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64080fbaf0 0x7f640810b340 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 --2- 172.21.15.59:0/2771012170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64080fcd00 0x7f640810b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 >> 172.21.15.59:0/2771012170 conn(0x7f64080f7190 msgr2=0x7f64080fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:14.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 shutdown_connections 2023-12-19T18:47:14.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.374+0000 7f640e983700 1 -- 172.21.15.59:0/2771012170 wait complete. 2023-12-19T18:47:14.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.488+0000 7efd118ae700 1 -- 172.21.15.59:0/387501061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd0c0fde00 msgr2=0x7efd0c0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:14.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.488+0000 7efd118ae700 1 --2- 172.21.15.59:0/387501061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd0c0fde00 0x7efd0c0fe250 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efd00004940 tx=0x7efd00009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:14.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.488+0000 7efd118ae700 1 -- 172.21.15.59:0/387501061 shutdown_connections 2023-12-19T18:47:14.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.488+0000 7efd118ae700 1 --2- 172.21.15.59:0/387501061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd0c0fde00 0x7efd0c0fe250 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.488+0000 7efd118ae700 1 --2- 172.21.15.59:0/387501061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd0c0fcbf0 0x7efd0c0fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.488+0000 7efd118ae700 1 -- 172.21.15.59:0/387501061 >> 172.21.15.59:0/387501061 conn(0x7efd0c0f82d0 msgr2=0x7efd0c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:14.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.488+0000 7efd118ae700 1 -- 172.21.15.59:0/387501061 shutdown_connections 2023-12-19T18:47:14.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.488+0000 7efd118ae700 1 -- 172.21.15.59:0/387501061 wait complete. 2023-12-19T18:47:14.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.489+0000 7efd118ae700 1 Processor -- start 2023-12-19T18:47:14.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.489+0000 7efd118ae700 1 -- start start 2023-12-19T18:47:14.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd118ae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd0c0fcbf0 0x7efd0c10d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd118ae700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd0c0fde00 0x7efd0c10da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd118ae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c10e030 con 0x7efd0c0fcbf0 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd118ae700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c10e8d0 con 0x7efd0c0fde00 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd0affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd0c0fcbf0 0x7efd0c10d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd0a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd0c0fde00 0x7efd0c10da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd0a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd0c0fde00 0x7efd0c10da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.59:50944/0 (socket says 172.21.15.59:50944) 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd0a7fc700 1 -- 172.21.15.59:0/1007916550 learned_addr learned my addr 172.21.15.59:0/1007916550 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd0affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd0c0fcbf0 0x7efd0c10d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58556/0 (socket says 172.21.15.59:58556) 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd0a7fc700 1 -- 172.21.15.59:0/1007916550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd0c0fcbf0 msgr2=0x7efd0c10d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd0a7fc700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd0c0fcbf0 0x7efd0c10d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd0a7fc700 1 -- 172.21.15.59:0/1007916550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00005040 con 0x7efd0c0fde00 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efd0a7fc700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd0c0fde00 0x7efd0c10da30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd00009fd0 tx=0x7efd00009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.490+0000 7efcf37fe700 1 -- 172.21.15.59:0/1007916550 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd0000e030 con 0x7efd0c0fde00 2023-12-19T18:47:14.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.491+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c10eb50 con 0x7efd0c0fde00 2023-12-19T18:47:14.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.491+0000 7efcf37fe700 1 -- 172.21.15.59:0/1007916550 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd00013c60 con 0x7efd0c0fde00 2023-12-19T18:47:14.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.491+0000 7efcf37fe700 1 -- 172.21.15.59:0/1007916550 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efd0001f070 con 0x7efd0c0fde00 2023-12-19T18:47:14.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.491+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c10f090 con 0x7efd0c0fde00 2023-12-19T18:47:14.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.492+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0c04bb80 con 0x7efd0c0fde00 2023-12-19T18:47:14.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.492+0000 7efcf37fe700 1 -- 172.21.15.59:0/1007916550 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7efd0001a570 con 0x7efd0c0fde00 2023-12-19T18:47:14.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.493+0000 7efcf37fe700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efcf4068f60 0x7efcf406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:14.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.493+0000 7efcf37fe700 1 -- 172.21.15.59:0/1007916550 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7efd0008c3b0 con 0x7efd0c0fde00 2023-12-19T18:47:14.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.493+0000 7efd0affd700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efcf4068f60 0x7efcf406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:14.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.494+0000 7efd0affd700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efcf4068f60 0x7efcf406b410 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009b80 tx=0x7efcfc00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:14.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.497+0000 7efcf37fe700 1 -- 172.21.15.59:0/1007916550 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7efd0005c2a0 con 0x7efd0c0fde00 2023-12-19T18:47:14.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.900+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efd0c105590 con 0x7efd0c0fde00 2023-12-19T18:47:14.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.900+0000 7efcf37fe700 1 -- 172.21.15.59:0/1007916550 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7efd0005be30 con 0x7efd0c0fde00 2023-12-19T18:47:14.902 INFO:teuthology.orchestra.run.smithi059.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2023-12-19T18:47:14.902 INFO:teuthology.orchestra.run.smithi059.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2023-12-19T18:47:14.902 INFO:teuthology.orchestra.run.smithi059.stdout: fs cephfs has deprecated feature inline_data enabled. 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efcf4068f60 msgr2=0x7efcf406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efcf4068f60 0x7efcf406b410 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009b80 tx=0x7efcfc00b040 comp rx=0 tx=0).stop 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd0c0fde00 msgr2=0x7efd0c10da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd0c0fde00 0x7efd0c10da30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd00009fd0 tx=0x7efd00009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 shutdown_connections 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efcf4068f60 0x7efcf406b410 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efd0c0fcbf0 0x7efd0c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 --2- 172.21.15.59:0/1007916550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd0c0fde00 0x7efd0c10da30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 >> 172.21.15.59:0/1007916550 conn(0x7efd0c0f82d0 msgr2=0x7efd0c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 shutdown_connections 2023-12-19T18:47:14.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:14.903+0000 7efd118ae700 1 -- 172.21.15.59:0/1007916550 wait complete. 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr:Note: switching to 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'. 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr: 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr:state without impacting any branches by switching back to a branch. 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr: 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr: 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr: git switch -c 2023-12-19T18:47:15.114 INFO:tasks.workunit.client.1.smithi089.stderr: 2023-12-19T18:47:15.115 INFO:tasks.workunit.client.1.smithi089.stderr:Or undo this operation with: 2023-12-19T18:47:15.115 INFO:tasks.workunit.client.1.smithi089.stderr: 2023-12-19T18:47:15.115 INFO:tasks.workunit.client.1.smithi089.stderr: git switch - 2023-12-19T18:47:15.115 INFO:tasks.workunit.client.1.smithi089.stderr: 2023-12-19T18:47:15.115 INFO:tasks.workunit.client.1.smithi089.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-19T18:47:15.115 INFO:tasks.workunit.client.1.smithi089.stderr: 2023-12-19T18:47:15.115 INFO:tasks.workunit.client.1.smithi089.stderr:HEAD is now at c7586e95519 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2023-12-15-0911 2023-12-19T18:47:15.120 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2023-12-19T18:47:15.179 INFO:tasks.workunit.client.1.smithi089.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-19T18:47:15.181 INFO:tasks.workunit.client.1.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-12-19T18:47:15.181 INFO:tasks.workunit.client.1.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-19T18:47:15.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:14 smithi059 ceph-mon[80661]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:15.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:14 smithi059 ceph-mon[80661]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T18:47:15.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:14 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1007916550' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-19T18:47:15.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:14 smithi089 ceph-mon[88459]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:15.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:14 smithi089 ceph-mon[88459]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T18:47:15.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:14 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1007916550' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-19T18:47:15.953 INFO:tasks.workunit.client.1.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-19T18:47:16.014 INFO:tasks.workunit.client.1.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-19T18:47:16.054 INFO:tasks.workunit.client.1.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-12-19T18:47:16.056 INFO:tasks.workunit.client.1.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-12-19T18:47:16.056 INFO:tasks.workunit.client.1.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-19T18:47:16.096 INFO:tasks.workunit.client.1.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-12-19T18:47:16.099 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T18:47:16.100 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2023-12-19T18:47:16.156 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2023-12-19T18:47:16.157 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2023-12-19T18:47:16.157 DEBUG:teuthology.orchestra.run.smithi089:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7586e95519b7be76f3b6e3a4b333888baf46bc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2023-12-19T18:47:16.217 INFO:tasks.workunit.client.1.smithi089.stderr:+ mkdir -p fsstress 2023-12-19T18:47:16.219 INFO:tasks.workunit.client.1.smithi089.stderr:+ pushd fsstress 2023-12-19T18:47:16.219 INFO:tasks.workunit.client.1.smithi089.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2023-12-19T18:47:16.219 INFO:tasks.workunit.client.1.smithi089.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2023-12-19T18:47:16.790 INFO:tasks.workunit.client.1.smithi089.stderr:+ tar xzf ltp-full.tgz 2023-12-19T18:47:17.886 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:17 smithi089 ceph-mon[88459]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T18:47:17.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:17 smithi059 ceph-mon[80661]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T18:47:19.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:19 smithi059 ceph-mon[80661]: pgmap v220: 65 pgs: 65 active+clean; 601 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 0 op/s 2023-12-19T18:47:20.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:19 smithi089 ceph-mon[88459]: pgmap v220: 65 pgs: 65 active+clean; 601 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 0 op/s 2023-12-19T18:47:20.530 INFO:tasks.workunit.client.0.smithi059.stderr:Updating files: 96% (11230/11620) Updating files: 97% (11272/11620) Updating files: 98% (11388/11620) Updating files: 99% (11504/11620) Updating files: 100% (11620/11620) Updating files: 100% (11620/11620), done. 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr:Note: switching to 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'. 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr: 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr:state without impacting any branches by switching back to a branch. 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr: 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr: 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr: git switch -c 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr: 2023-12-19T18:47:21.451 INFO:tasks.workunit.client.0.smithi059.stderr:Or undo this operation with: 2023-12-19T18:47:21.452 INFO:tasks.workunit.client.0.smithi059.stderr: 2023-12-19T18:47:21.452 INFO:tasks.workunit.client.0.smithi059.stderr: git switch - 2023-12-19T18:47:21.452 INFO:tasks.workunit.client.0.smithi059.stderr: 2023-12-19T18:47:21.452 INFO:tasks.workunit.client.0.smithi059.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-19T18:47:21.452 INFO:tasks.workunit.client.0.smithi059.stderr: 2023-12-19T18:47:21.452 INFO:tasks.workunit.client.0.smithi059.stderr:HEAD is now at c7586e95519 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2023-12-15-0911 2023-12-19T18:47:21.456 DEBUG:teuthology.orchestra.run.smithi059:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-19T18:47:21.485 INFO:tasks.workunit.client.0.smithi059.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-19T18:47:21.487 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-19T18:47:21.487 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-19T18:47:21.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:21 smithi059 ceph-mon[80661]: pgmap v221: 65 pgs: 65 active+clean; 601 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 0 op/s 2023-12-19T18:47:22.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:21 smithi089 ceph-mon[88459]: pgmap v221: 65 pgs: 65 active+clean; 601 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s wr, 0 op/s 2023-12-19T18:47:22.389 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-19T18:47:22.491 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-19T18:47:22.560 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-19T18:47:22.562 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-19T18:47:22.563 INFO:tasks.workunit.client.0.smithi059.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-19T18:47:22.614 INFO:tasks.workunit.client.0.smithi059.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-19T18:47:22.624 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:47:22.624 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-19T18:47:22.682 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2023-12-19T18:47:22.684 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2023-12-19T18:47:22.684 DEBUG:teuthology.orchestra.run.smithi059:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7586e95519b7be76f3b6e3a4b333888baf46bc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2023-12-19T18:47:22.749 INFO:tasks.workunit.client.0.smithi059.stderr:+ mkdir -p fsstress 2023-12-19T18:47:22.752 INFO:tasks.workunit.client.0.smithi059.stderr:+ pushd fsstress 2023-12-19T18:47:22.752 INFO:tasks.workunit.client.0.smithi059.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2023-12-19T18:47:22.753 INFO:tasks.workunit.client.0.smithi059.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2023-12-19T18:47:23.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:22 smithi089 ceph-mon[88459]: pgmap v222: 65 pgs: 65 active+clean; 9.2 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 763 KiB/s wr, 42 op/s 2023-12-19T18:47:23.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:22 smithi059 ceph-mon[80661]: pgmap v222: 65 pgs: 65 active+clean; 9.2 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 763 KiB/s wr, 42 op/s 2023-12-19T18:47:24.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:23 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:47:24.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:23 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:47:25.006 INFO:tasks.workunit.client.0.smithi059.stderr:+ tar xzf ltp-full.tgz 2023-12-19T18:47:25.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:24 smithi089 ceph-mon[88459]: pgmap v223: 65 pgs: 65 active+clean; 24 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s wr, 114 op/s 2023-12-19T18:47:25.448 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:24 smithi059 ceph-mon[80661]: pgmap v223: 65 pgs: 65 active+clean; 24 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s wr, 114 op/s 2023-12-19T18:47:27.887 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:27 smithi089 ceph-mon[88459]: pgmap v224: 65 pgs: 65 active+clean; 48 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.2 MiB/s wr, 408 op/s 2023-12-19T18:47:27.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:27 smithi059 ceph-mon[80661]: pgmap v224: 65 pgs: 65 active+clean; 48 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.2 MiB/s wr, 408 op/s 2023-12-19T18:47:29.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:29 smithi059 ceph-mon[80661]: pgmap v225: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 455 op/s 2023-12-19T18:47:30.004 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:29 smithi089 ceph-mon[88459]: pgmap v225: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 4.9 MiB/s wr, 455 op/s 2023-12-19T18:47:31.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:31 smithi059 ceph-mon[80661]: pgmap v226: 65 pgs: 65 active+clean; 73 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 558 op/s 2023-12-19T18:47:32.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:31 smithi089 ceph-mon[88459]: pgmap v226: 65 pgs: 65 active+clean; 73 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 558 op/s 2023-12-19T18:47:33.254 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:32 smithi089 ceph-mon[88459]: pgmap v227: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.6 MiB/s wr, 602 op/s 2023-12-19T18:47:33.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:32 smithi059 ceph-mon[80661]: pgmap v227: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 7.6 MiB/s wr, 602 op/s 2023-12-19T18:47:35.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:35 smithi089 ceph-mon[88459]: pgmap v228: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.9 MiB/s wr, 560 op/s 2023-12-19T18:47:35.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:35 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:35.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:35 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.isaaav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T18:47:35.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:35 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T18:47:35.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:35 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:47:35.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:35 smithi059 ceph-mon[80661]: pgmap v228: 65 pgs: 65 active+clean; 87 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.9 MiB/s wr, 560 op/s 2023-12-19T18:47:35.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:35 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:35.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:35 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi089.isaaav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T18:47:35.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:35 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T18:47:35.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:35 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T18:47:36.784 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:36 smithi089 ceph-mon[88459]: Upgrade: Updating mgr.smithi089.isaaav 2023-12-19T18:47:36.784 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:36 smithi089 ceph-mon[88459]: Deploying daemon mgr.smithi089.isaaav on smithi089 2023-12-19T18:47:36.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:36 smithi059 ceph-mon[80661]: Upgrade: Updating mgr.smithi089.isaaav 2023-12-19T18:47:36.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:36 smithi059 ceph-mon[80661]: Deploying daemon mgr.smithi089.isaaav on smithi089 2023-12-19T18:47:37.889 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:37 smithi089 ceph-mon[88459]: pgmap v229: 65 pgs: 65 active+clean; 95 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.3 MiB/s wr, 493 op/s 2023-12-19T18:47:37.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:37 smithi059 ceph-mon[80661]: pgmap v229: 65 pgs: 65 active+clean; 95 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.3 MiB/s wr, 493 op/s 2023-12-19T18:47:38.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:38 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:47:39.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:38 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T18:47:39.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:39 smithi059 ceph-mon[80661]: pgmap v230: 65 pgs: 65 active+clean; 99 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.4 MiB/s wr, 199 op/s 2023-12-19T18:47:40.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:39 smithi089 ceph-mon[88459]: pgmap v230: 65 pgs: 65 active+clean; 99 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 4.4 MiB/s wr, 199 op/s 2023-12-19T18:47:41.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:41 smithi059 ceph-mon[80661]: pgmap v231: 65 pgs: 65 active+clean; 136 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 7.0 MiB/s wr, 582 op/s 2023-12-19T18:47:42.005 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:41 smithi089 ceph-mon[88459]: pgmap v231: 65 pgs: 65 active+clean; 136 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 7.0 MiB/s wr, 582 op/s 2023-12-19T18:47:43.255 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:42 smithi089 ceph-mon[88459]: pgmap v232: 65 pgs: 65 active+clean; 152 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.0 MiB/s wr, 707 op/s 2023-12-19T18:47:43.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:42 smithi059 ceph-mon[80661]: pgmap v232: 65 pgs: 65 active+clean; 152 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.0 MiB/s wr, 707 op/s 2023-12-19T18:47:45.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.052+0000 7f3490c27700 1 -- 172.21.15.59:0/903320036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fdb60 msgr2=0x7f348c104420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.052+0000 7f3490c27700 1 --2- 172.21.15.59:0/903320036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fdb60 0x7f348c104420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3480004940 tx=0x7f3480009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:45.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.053+0000 7f3490c27700 1 -- 172.21.15.59:0/903320036 shutdown_connections 2023-12-19T18:47:45.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.053+0000 7f3490c27700 1 --2- 172.21.15.59:0/903320036 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fdb60 0x7f348c104420 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.053+0000 7f3490c27700 1 --2- 172.21.15.59:0/903320036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c0fd180 0x7f348c0fd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.053+0000 7f3490c27700 1 -- 172.21.15.59:0/903320036 >> 172.21.15.59:0/903320036 conn(0x7f348c0f6950 msgr2=0x7f348c0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:45.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.053+0000 7f3490c27700 1 -- 172.21.15.59:0/903320036 shutdown_connections 2023-12-19T18:47:45.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.053+0000 7f3490c27700 1 -- 172.21.15.59:0/903320036 wait complete. 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.053+0000 7f3490c27700 1 Processor -- start 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.053+0000 7f3490c27700 1 -- start start 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f3490c27700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fd180 0x7f348c0fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f3490c27700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c0fe8f0 0x7f348c0ff7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f3490c27700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0fedb0 con 0x7f348c0fd180 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f3490c27700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c0fef20 con 0x7f348c0fe8f0 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f348a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fd180 0x7f348c0fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f348a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fd180 0x7f348c0fe3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45958/0 (socket says 172.21.15.59:45958) 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f348a59c700 1 -- 172.21.15.59:0/2868605164 learned_addr learned my addr 172.21.15.59:0/2868605164 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:45.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f3489d9b700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c0fe8f0 0x7f348c0ff7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:45.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f348a59c700 1 -- 172.21.15.59:0/2868605164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c0fe8f0 msgr2=0x7f348c0ff7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f348a59c700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c0fe8f0 0x7f348c0ff7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.054+0000 7f348a59c700 1 -- 172.21.15.59:0/2868605164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3480005040 con 0x7f348c0fd180 2023-12-19T18:47:45.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.055+0000 7f348a59c700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fd180 0x7f348c0fe3b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f347c002a50 tx=0x7f347c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:45.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.055+0000 7f347affd700 1 -- 172.21.15.59:0/2868605164 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347c007cc0 con 0x7f348c0fd180 2023-12-19T18:47:45.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.055+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348c0ffe40 con 0x7f348c0fd180 2023-12-19T18:47:45.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.055+0000 7f347affd700 1 -- 172.21.15.59:0/2868605164 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f347c00f040 con 0x7f348c0fd180 2023-12-19T18:47:45.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.055+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348c100330 con 0x7f348c0fd180 2023-12-19T18:47:45.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.055+0000 7f347affd700 1 -- 172.21.15.59:0/2868605164 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f347c00d9c0 con 0x7f348c0fd180 2023-12-19T18:47:45.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.056+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348c109830 con 0x7f348c0fd180 2023-12-19T18:47:45.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.057+0000 7f347affd700 1 -- 172.21.15.59:0/2868605164 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f347c00dc20 con 0x7f348c0fd180 2023-12-19T18:47:45.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.057+0000 7f347affd700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3474068f60 0x7f347406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:45.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.057+0000 7f347affd700 1 -- 172.21.15.59:0/2868605164 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f347c0863d0 con 0x7f348c0fd180 2023-12-19T18:47:45.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.058+0000 7f3489d9b700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3474068f60 0x7f347406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:45.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.058+0000 7f3489d9b700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3474068f60 0x7f347406b410 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3480004890 tx=0x7f3480009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:45.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.066+0000 7f347affd700 1 -- 172.21.15.59:0/2868605164 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f347c04ca00 con 0x7f348c0fd180 2023-12-19T18:47:45.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.312+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f348c1079a0 con 0x7f3474068f60 2023-12-19T18:47:45.314 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.313+0000 7f347affd700 1 -- 172.21.15.59:0/2868605164 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+328 (secure 0 0 0) 0x7f3480007e00 con 0x7f3474068f60 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.315+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3474068f60 msgr2=0x7f347406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.315+0000 7f3490c27700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3474068f60 0x7f347406b410 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3480004890 tx=0x7f3480009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.315+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fd180 msgr2=0x7f348c0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.315+0000 7f3490c27700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fd180 0x7f348c0fe3b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f347c002a50 tx=0x7f347c0046b0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.315+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 shutdown_connections 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.315+0000 7f3490c27700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f3474068f60 0x7f347406b410 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.315+0000 7f3490c27700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f348c0fd180 0x7f348c0fe3b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.315+0000 7f3490c27700 1 --2- 172.21.15.59:0/2868605164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f348c0fe8f0 0x7f348c0ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.315+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 >> 172.21.15.59:0/2868605164 conn(0x7f348c0f6950 msgr2=0x7f348c0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.316+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 shutdown_connections 2023-12-19T18:47:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.316+0000 7f3490c27700 1 -- 172.21.15.59:0/2868605164 wait complete. 2023-12-19T18:47:45.328 INFO:teuthology.orchestra.run.smithi059.stdout:true 2023-12-19T18:47:45.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.457+0000 7f9634b32700 1 -- 172.21.15.59:0/1815429800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 msgr2=0x7f96300fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.457+0000 7f9634b32700 1 --2- 172.21.15.59:0/1815429800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 0x7f96300fe250 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9624004940 tx=0x7f9624009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:45.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.457+0000 7f9634b32700 1 -- 172.21.15.59:0/1815429800 shutdown_connections 2023-12-19T18:47:45.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.457+0000 7f9634b32700 1 --2- 172.21.15.59:0/1815429800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 0x7f96300fe250 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.457+0000 7f9634b32700 1 --2- 172.21.15.59:0/1815429800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300fcbf0 0x7f96300fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.457+0000 7f9634b32700 1 -- 172.21.15.59:0/1815429800 >> 172.21.15.59:0/1815429800 conn(0x7f96300f82d0 msgr2=0x7f96300fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:45.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.457+0000 7f9634b32700 1 -- 172.21.15.59:0/1815429800 shutdown_connections 2023-12-19T18:47:45.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.457+0000 7f9634b32700 1 -- 172.21.15.59:0/1815429800 wait complete. 2023-12-19T18:47:45.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f9634b32700 1 Processor -- start 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f9634b32700 1 -- start start 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f9634b32700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300fcbf0 0x7f963010d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f9634b32700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 0x7f963010dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f9634b32700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963010e0c0 con 0x7f96300fde00 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f9634b32700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963010e960 con 0x7f96300fcbf0 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f962dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 0x7f963010dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f962e59c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300fcbf0 0x7f963010d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f962dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 0x7f963010dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45980/0 (socket says 172.21.15.59:45980) 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.458+0000 7f962dd9b700 1 -- 172.21.15.59:0/861757861 learned_addr learned my addr 172.21.15.59:0/861757861 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f962e59c700 1 -- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 msgr2=0x7f963010dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f962e59c700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 0x7f963010dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f962e59c700 1 -- 172.21.15.59:0/861757861 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9624005040 con 0x7f96300fcbf0 2023-12-19T18:47:45.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f962dd9b700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 0x7f963010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:47:45.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f962e59c700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300fcbf0 0x7f963010d580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9620002840 tx=0x7f9620004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:45.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f961effd700 1 -- 172.21.15.59:0/861757861 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9620007cc0 con 0x7f96300fcbf0 2023-12-19T18:47:45.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963010ec40 con 0x7f96300fcbf0 2023-12-19T18:47:45.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f961effd700 1 -- 172.21.15.59:0/861757861 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f962000f040 con 0x7f96300fcbf0 2023-12-19T18:47:45.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f961effd700 1 -- 172.21.15.59:0/861757861 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9620013860 con 0x7f96300fcbf0 2023-12-19T18:47:45.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.459+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963010f180 con 0x7f96300fcbf0 2023-12-19T18:47:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.461+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963004bb80 con 0x7f96300fcbf0 2023-12-19T18:47:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.461+0000 7f961effd700 1 -- 172.21.15.59:0/861757861 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f962001c400 con 0x7f96300fcbf0 2023-12-19T18:47:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.461+0000 7f961effd700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9618068f70 0x7f961806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.461+0000 7f961effd700 1 -- 172.21.15.59:0/861757861 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9620086a20 con 0x7f96300fcbf0 2023-12-19T18:47:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.461+0000 7f962dd9b700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9618068f70 0x7f961806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:45.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.462+0000 7f962dd9b700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9618068f70 0x7f961806b420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f962400b3b0 tx=0x7f9624009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:45.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.466+0000 7f961effd700 1 -- 172.21.15.59:0/861757861 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f96200568d0 con 0x7f96300fcbf0 2023-12-19T18:47:45.657 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:45 smithi089 ceph-mon[88459]: pgmap v233: 65 pgs: 65 active+clean; 159 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 733 op/s 2023-12-19T18:47:45.657 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:45.658 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:45.658 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:45 smithi089 ceph-mon[88459]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:47:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.735+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f963010f5e0 con 0x7f9618068f70 2023-12-19T18:47:45.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.737+0000 7f961effd700 1 -- 172.21.15.59:0/861757861 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+328 (secure 0 0 0) 0x7f963010f5e0 con 0x7f9618068f70 2023-12-19T18:47:45.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9618068f70 msgr2=0x7f961806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9618068f70 0x7f961806b420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f962400b3b0 tx=0x7f9624009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:45.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300fcbf0 msgr2=0x7f963010d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300fcbf0 0x7f963010d580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9620002840 tx=0x7f9620004570 comp rx=0 tx=0).stop 2023-12-19T18:47:45.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 shutdown_connections 2023-12-19T18:47:45.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96300fcbf0 0x7f963010d580 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f9618068f70 0x7f961806b420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 --2- 172.21.15.59:0/861757861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f96300fde00 0x7f963010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 >> 172.21.15.59:0/861757861 conn(0x7f96300f82d0 msgr2=0x7f9630101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:45.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 shutdown_connections 2023-12-19T18:47:45.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.739+0000 7f9634b32700 1 -- 172.21.15.59:0/861757861 wait complete. 2023-12-19T18:47:45.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.865+0000 7f7fea08d700 1 -- 172.21.15.59:0/3320636899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fcbf0 msgr2=0x7f7fe40fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.865+0000 7f7fea08d700 1 --2- 172.21.15.59:0/3320636899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fcbf0 0x7f7fe40fd000 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4004940 tx=0x7f7fd4009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:45.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.865+0000 7f7fea08d700 1 -- 172.21.15.59:0/3320636899 shutdown_connections 2023-12-19T18:47:45.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.865+0000 7f7fea08d700 1 --2- 172.21.15.59:0/3320636899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fe40fde00 0x7f7fe40fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.865+0000 7f7fea08d700 1 --2- 172.21.15.59:0/3320636899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fcbf0 0x7f7fe40fd000 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.865+0000 7f7fea08d700 1 -- 172.21.15.59:0/3320636899 >> 172.21.15.59:0/3320636899 conn(0x7f7fe40f82d0 msgr2=0x7f7fe40fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:45.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.865+0000 7f7fea08d700 1 -- 172.21.15.59:0/3320636899 shutdown_connections 2023-12-19T18:47:45.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.865+0000 7f7fea08d700 1 -- 172.21.15.59:0/3320636899 wait complete. 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.866+0000 7f7fea08d700 1 Processor -- start 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.866+0000 7f7fea08d700 1 -- start start 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.866+0000 7f7fea08d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fe40fcbf0 0x7f7fe410f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.866+0000 7f7fea08d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fde00 0x7f7fe410fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.866+0000 7f7fea08d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe4110220 con 0x7f7fe40fde00 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.866+0000 7f7fea08d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe4110ac0 con 0x7f7fe40fcbf0 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.866+0000 7f7fe2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fde00 0x7f7fe410fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.866+0000 7f7fe2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fde00 0x7f7fe410fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:45994/0 (socket says 172.21.15.59:45994) 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.866+0000 7f7fe2ffd700 1 -- 172.21.15.59:0/2002612183 learned_addr learned my addr 172.21.15.59:0/2002612183 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:45.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fe37fe700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fe40fcbf0 0x7f7fe410f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:45.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fe2ffd700 1 -- 172.21.15.59:0/2002612183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fe40fcbf0 msgr2=0x7f7fe410f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:45.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fe2ffd700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fe40fcbf0 0x7f7fe410f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:45.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fe2ffd700 1 -- 172.21.15.59:0/2002612183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd4005040 con 0x7f7fe40fde00 2023-12-19T18:47:45.869 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:45 smithi059 ceph-mon[80661]: pgmap v233: 65 pgs: 65 active+clean; 159 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 733 op/s 2023-12-19T18:47:45.869 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:45.869 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' 2023-12-19T18:47:45.869 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:45 smithi059 ceph-mon[80661]: from='mgr.14210 172.21.15.59:0/2325809785' entity='mgr.smithi059.ccbnob' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T18:47:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fe37fe700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fe40fcbf0 0x7f7fe410f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:47:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fe2ffd700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fde00 0x7f7fe410fc20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8002a80 tx=0x7f7fd80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fcbfff700 1 -- 172.21.15.59:0/2002612183 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fd8007cc0 con 0x7f7fe40fde00 2023-12-19T18:47:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe4110da0 con 0x7f7fe40fde00 2023-12-19T18:47:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fcbfff700 1 -- 172.21.15.59:0/2002612183 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fd800f040 con 0x7f7fe40fde00 2023-12-19T18:47:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fcbfff700 1 -- 172.21.15.59:0/2002612183 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fd800da60 con 0x7f7fe40fde00 2023-12-19T18:47:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.867+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe41112e0 con 0x7f7fe40fde00 2023-12-19T18:47:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.869+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe4101ea0 con 0x7f7fe40fde00 2023-12-19T18:47:45.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.870+0000 7f7fcbfff700 1 -- 172.21.15.59:0/2002612183 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7fd800dbc0 con 0x7f7fe40fde00 2023-12-19T18:47:45.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.870+0000 7f7fcbfff700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7fcc068fb0 0x7f7fcc06b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:45.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.870+0000 7f7fcbfff700 1 -- 172.21.15.59:0/2002612183 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7fd8086c10 con 0x7f7fe40fde00 2023-12-19T18:47:45.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.870+0000 7f7fe37fe700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7fcc068fb0 0x7f7fcc06b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:45.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.871+0000 7f7fe37fe700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7fcc068fb0 0x7f7fcc06b460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4004890 tx=0x7f7fd4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:45.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:45.876+0000 7f7fcbfff700 1 -- 172.21.15.59:0/2002612183 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7fd8056a00 con 0x7f7fe40fde00 2023-12-19T18:47:46.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.156+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fe4111610 con 0x7f7fcc068fb0 2023-12-19T18:47:46.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.175+0000 7f7fcbfff700 1 -- 172.21.15.59:0/2002612183 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f7fe4111610 con 0x7f7fcc068fb0 2023-12-19T18:47:46.176 INFO:teuthology.orchestra.run.smithi059.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-19T18:47:46.176 INFO:teuthology.orchestra.run.smithi059.stdout:alertmanager.smithi059 smithi059 *:9093,9094 running (5m) 90s ago 7m 25.1M - 0.25.0 c8568f914cd2 528f56ce4fdd 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi059 smithi059 running (7m) 90s ago 7m 6660k - 17.2.7-131-ga88f3edf f9ecdbaa1030 03ae34e0a398 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:crash.smithi089 smithi089 running (6m) 98s ago 6m 6665k - 17.2.7-131-ga88f3edf f9ecdbaa1030 8a39e517426d 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:grafana.smithi059 smithi059 *:3000 running (5m) 90s ago 7m 87.0M - 9.4.7 954c08fa6188 a19de18455de 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.abqklt smithi059 running (117s) 90s ago 116s 10.4M - 17.2.7-131-ga88f3edf f9ecdbaa1030 c303f5a4640b 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi059.rtqzmc smithi059 running (110s) 90s ago 109s 14.2M - 17.2.7-131-ga88f3edf f9ecdbaa1030 a2c3ca2a9ed2 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi089.dfrkla smithi089 running (107s) 98s ago 107s 14.5M - 17.2.7-131-ga88f3edf f9ecdbaa1030 f707fc4d33e4 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:mds.cephfs.smithi089.dpgsbt smithi089 running (114s) 98s ago 114s 12.1M - 17.2.7-131-ga88f3edf f9ecdbaa1030 ce89c8fa8ca4 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi059.ccbnob smithi059 *:9283 running (8m) 90s ago 8m 479M - 17.2.7-131-ga88f3edf f9ecdbaa1030 9ce46f56adc1 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:mgr.smithi089.isaaav smithi089 *:8443,9283 starting - - - - 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi059 smithi059 running (8m) 90s ago 8m 45.1M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 297c7051711f 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:mon.smithi089 smithi089 running (6m) 98s ago 6m 33.0M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 5ac855c460fa 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi059 smithi059 *:9100 running (7m) 90s ago 7m 16.0M - 1.5.0 0da6a335fe13 bfe5bff75fbd 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:node-exporter.smithi089 smithi089 *:9100 running (6m) 98s ago 6m 15.7M - 1.5.0 0da6a335fe13 057c3879e7a2 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:osd.0 smithi059 running (5m) 90s ago 5m 48.2M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 87e75bce1c5b 2023-12-19T18:47:46.177 INFO:teuthology.orchestra.run.smithi059.stdout:osd.1 smithi059 running (5m) 90s ago 5m 48.5M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 1b133ef58db0 2023-12-19T18:47:46.178 INFO:teuthology.orchestra.run.smithi059.stdout:osd.2 smithi059 running (4m) 90s ago 4m 45.0M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 0f56936ea44c 2023-12-19T18:47:46.178 INFO:teuthology.orchestra.run.smithi059.stdout:osd.3 smithi089 running (4m) 98s ago 4m 44.7M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 3eafaf27a901 2023-12-19T18:47:46.178 INFO:teuthology.orchestra.run.smithi059.stdout:osd.4 smithi089 running (3m) 98s ago 3m 44.0M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 cc630810bb79 2023-12-19T18:47:46.178 INFO:teuthology.orchestra.run.smithi059.stdout:osd.5 smithi089 running (3m) 98s ago 3m 41.3M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 f1d513a4069b 2023-12-19T18:47:46.178 INFO:teuthology.orchestra.run.smithi059.stdout:prometheus.smithi059 smithi059 *:9095 running (5m) 90s ago 6m 52.1M - 2.43.0 a07b618ecd1d 5884a00b2f95 2023-12-19T18:47:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7fcc068fb0 msgr2=0x7f7fcc06b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7fcc068fb0 0x7f7fcc06b460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4004890 tx=0x7f7fd4009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fde00 msgr2=0x7f7fe410fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fde00 0x7f7fe410fc20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8002a80 tx=0x7f7fd80046b0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 shutdown_connections 2023-12-19T18:47:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7fe40fcbf0 0x7f7fe410f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7fcc068fb0 0x7f7fcc06b460 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 --2- 172.21.15.59:0/2002612183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7fe40fde00 0x7f7fe410fc20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 >> 172.21.15.59:0/2002612183 conn(0x7f7fe40f82d0 msgr2=0x7f7fe40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 shutdown_connections 2023-12-19T18:47:46.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.178+0000 7f7fea08d700 1 -- 172.21.15.59:0/2002612183 wait complete. 2023-12-19T18:47:46.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.292+0000 7efc6d8ff700 1 -- 172.21.15.59:0/4082997222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc680fd730 msgr2=0x7efc680ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:46.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.292+0000 7efc6d8ff700 1 --2- 172.21.15.59:0/4082997222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc680fd730 0x7efc680ffb10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7efc58004940 tx=0x7efc58009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:46.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.292+0000 7efc6d8ff700 1 -- 172.21.15.59:0/4082997222 shutdown_connections 2023-12-19T18:47:46.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.292+0000 7efc6d8ff700 1 --2- 172.21.15.59:0/4082997222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc68100050 0x7efc68102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.292+0000 7efc6d8ff700 1 --2- 172.21.15.59:0/4082997222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc680fd730 0x7efc680ffb10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.292+0000 7efc6d8ff700 1 -- 172.21.15.59:0/4082997222 >> 172.21.15.59:0/4082997222 conn(0x7efc680f7190 msgr2=0x7efc680f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:46.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.292+0000 7efc6d8ff700 1 -- 172.21.15.59:0/4082997222 shutdown_connections 2023-12-19T18:47:46.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.292+0000 7efc6d8ff700 1 -- 172.21.15.59:0/4082997222 wait complete. 2023-12-19T18:47:46.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.293+0000 7efc6d8ff700 1 Processor -- start 2023-12-19T18:47:46.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.293+0000 7efc6d8ff700 1 -- start start 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.293+0000 7efc6d8ff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc680fd730 0x7efc6810d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.293+0000 7efc6d8ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc68100050 0x7efc6810d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.293+0000 7efc6d8ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6810dfd0 con 0x7efc68100050 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.293+0000 7efc6d8ff700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6810e870 con 0x7efc680fd730 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.293+0000 7efc667fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc68100050 0x7efc6810d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.293+0000 7efc667fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc68100050 0x7efc6810d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46012/0 (socket says 172.21.15.59:46012) 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.293+0000 7efc667fc700 1 -- 172.21.15.59:0/394964430 learned_addr learned my addr 172.21.15.59:0/394964430 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc66ffd700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc680fd730 0x7efc6810d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc667fc700 1 -- 172.21.15.59:0/394964430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc680fd730 msgr2=0x7efc6810d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc667fc700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc680fd730 0x7efc6810d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc667fc700 1 -- 172.21.15.59:0/394964430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc58005040 con 0x7efc68100050 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc66ffd700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc680fd730 0x7efc6810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc667fc700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc68100050 0x7efc6810d9d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00eb40 tx=0x7efc5c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:46.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc4f7fe700 1 -- 172.21.15.59:0/394964430 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc5c007cc0 con 0x7efc68100050 2023-12-19T18:47:46.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc4f7fe700 1 -- 172.21.15.59:0/394964430 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc5c00f040 con 0x7efc68100050 2023-12-19T18:47:46.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6810eaf0 con 0x7efc68100050 2023-12-19T18:47:46.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc4f7fe700 1 -- 172.21.15.59:0/394964430 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc5c00c990 con 0x7efc68100050 2023-12-19T18:47:46.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.294+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6810f030 con 0x7efc68100050 2023-12-19T18:47:46.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.295+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6804bb80 con 0x7efc68100050 2023-12-19T18:47:46.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.296+0000 7efc4f7fe700 1 -- 172.21.15.59:0/394964430 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7efc5c00caf0 con 0x7efc68100050 2023-12-19T18:47:46.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.297+0000 7efc4f7fe700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efc50068f60 0x7efc5006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:46.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.297+0000 7efc4f7fe700 1 -- 172.21.15.59:0/394964430 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7efc5c0869c0 con 0x7efc68100050 2023-12-19T18:47:46.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.297+0000 7efc66ffd700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efc50068f60 0x7efc5006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:46.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.298+0000 7efc66ffd700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efc50068f60 0x7efc5006b410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efc58000c00 tx=0x7efc58009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:46.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.301+0000 7efc4f7fe700 1 -- 172.21.15.59:0/394964430 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7efc5c04da00 con 0x7efc68100050 2023-12-19T18:47:46.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.700+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7efc68105580 con 0x7efc68100050 2023-12-19T18:47:46.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.700+0000 7efc4f7fe700 1 -- 172.21.15.59:0/394964430 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7efc5c0568e0 con 0x7efc68100050 2023-12-19T18:47:46.702 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-12-19T18:47:46.703 INFO:teuthology.orchestra.run.smithi059.stdout: "mon": { 2023-12-19T18:47:46.703 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 2 2023-12-19T18:47:46.703 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:47:46.703 INFO:teuthology.orchestra.run.smithi059.stdout: "mgr": { 2023-12-19T18:47:46.703 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 2 2023-12-19T18:47:46.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:47:46.704 INFO:teuthology.orchestra.run.smithi059.stdout: "osd": { 2023-12-19T18:47:46.704 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 6 2023-12-19T18:47:46.704 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:47:46.704 INFO:teuthology.orchestra.run.smithi059.stdout: "mds": { 2023-12-19T18:47:46.705 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 4 2023-12-19T18:47:46.705 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2023-12-19T18:47:46.705 INFO:teuthology.orchestra.run.smithi059.stdout: "overall": { 2023-12-19T18:47:46.705 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 14 2023-12-19T18:47:46.705 INFO:teuthology.orchestra.run.smithi059.stdout: } 2023-12-19T18:47:46.705 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-12-19T18:47:46.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efc50068f60 msgr2=0x7efc5006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:46.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efc50068f60 0x7efc5006b410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efc58000c00 tx=0x7efc58009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:46.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc68100050 msgr2=0x7efc6810d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:46.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc68100050 0x7efc6810d9d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00eb40 tx=0x7efc5c00b040 comp rx=0 tx=0).stop 2023-12-19T18:47:46.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 shutdown_connections 2023-12-19T18:47:46.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc680fd730 0x7efc6810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7efc50068f60 0x7efc5006b410 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 --2- 172.21.15.59:0/394964430 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc68100050 0x7efc6810d9d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 >> 172.21.15.59:0/394964430 conn(0x7efc680f7190 msgr2=0x7efc680f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:46.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 shutdown_connections 2023-12-19T18:47:46.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.703+0000 7efc6d8ff700 1 -- 172.21.15.59:0/394964430 wait complete. 2023-12-19T18:47:46.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 -- 172.21.15.59:0/913773880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fbaf0 msgr2=0x7fafd00fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:46.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 --2- 172.21.15.59:0/913773880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fbaf0 0x7fafd00fbf00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fafcc004940 tx=0x7fafcc009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:46.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 -- 172.21.15.59:0/913773880 shutdown_connections 2023-12-19T18:47:46.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 --2- 172.21.15.59:0/913773880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00fcd00 0x7fafd00fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 --2- 172.21.15.59:0/913773880 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fbaf0 0x7fafd00fbf00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 -- 172.21.15.59:0/913773880 >> 172.21.15.59:0/913773880 conn(0x7fafd00f7190 msgr2=0x7fafd00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:46.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 -- 172.21.15.59:0/913773880 shutdown_connections 2023-12-19T18:47:46.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 -- 172.21.15.59:0/913773880 wait complete. 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 Processor -- start 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.820+0000 7fafd76d3700 1 -- start start 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd76d3700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00fbaf0 0x7fafd010f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd76d3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fcd00 0x7fafd010fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd76d3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd01102c0 con 0x7fafd00fcd00 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd76d3700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd0110b60 con 0x7fafd00fbaf0 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd4c6e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fcd00 0x7fafd010fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd4c6e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fcd00 0x7fafd010fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46030/0 (socket says 172.21.15.59:46030) 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd4c6e700 1 -- 172.21.15.59:0/1291635298 learned_addr learned my addr 172.21.15.59:0/1291635298 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:46.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd546f700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00fbaf0 0x7fafd010f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd4c6e700 1 -- 172.21.15.59:0/1291635298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00fbaf0 msgr2=0x7fafd010f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd4c6e700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00fbaf0 0x7fafd010f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd4c6e700 1 -- 172.21.15.59:0/1291635298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc005040 con 0x7fafd00fcd00 2023-12-19T18:47:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.821+0000 7fafd4c6e700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fcd00 0x7fafd010fcc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fafc0002a50 tx=0x7fafc00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.822+0000 7fafc5ffb700 1 -- 172.21.15.59:0/1291635298 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafc0007cc0 con 0x7fafd00fcd00 2023-12-19T18:47:46.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.822+0000 7fafc5ffb700 1 -- 172.21.15.59:0/1291635298 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafc000f040 con 0x7fafd00fcd00 2023-12-19T18:47:46.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.822+0000 7fafc5ffb700 1 -- 172.21.15.59:0/1291635298 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafc000d9c0 con 0x7fafd00fcd00 2023-12-19T18:47:46.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.822+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd0110e40 con 0x7fafd00fcd00 2023-12-19T18:47:46.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.822+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd0111380 con 0x7fafd00fcd00 2023-12-19T18:47:46.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.823+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd004bb80 con 0x7fafd00fcd00 2023-12-19T18:47:46.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.824+0000 7fafc5ffb700 1 -- 172.21.15.59:0/1291635298 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fafc000db20 con 0x7fafd00fcd00 2023-12-19T18:47:46.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.824+0000 7fafc5ffb700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fafbc068f60 0x7fafbc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:46.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.824+0000 7fafc5ffb700 1 -- 172.21.15.59:0/1291635298 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fafc0086a30 con 0x7fafd00fcd00 2023-12-19T18:47:46.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.824+0000 7fafd546f700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fafbc068f60 0x7fafbc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:46.826 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.825+0000 7fafd546f700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fafbc068f60 0x7fafbc06b410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fafcc004890 tx=0x7fafcc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:46.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:46.829+0000 7fafc5ffb700 1 -- 172.21.15.59:0/1291635298 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fafc0056940 con 0x7fafd00fcd00 2023-12-19T18:47:46.865 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:46 smithi089 ceph-mon[88459]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:46.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:46 smithi059 ceph-mon[80661]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:47.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.166+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fafd0104490 con 0x7fafd00fcd00 2023-12-19T18:47:47.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.166+0000 7fafc5ffb700 1 -- 172.21.15.59:0/1291635298 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1805 (secure 0 0 0) 0x7fafc00564d0 con 0x7fafd00fcd00 2023-12-19T18:47:47.167 INFO:teuthology.orchestra.run.smithi059.stdout:e15 2023-12-19T18:47:47.167 INFO:teuthology.orchestra.run.smithi059.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:legacy client fscid: 1 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:Filesystem 'cephfs' (1) 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:fs_name cephfs 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 15 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:created 2023-12-19T18:45:45.631056+0000 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:modified 2023-12-19T18:46:03.615908+0000 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:tableserver 0 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:root 0 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:session_timeout 60 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:session_autoclose 300 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:max_file_size 1099511627776 2023-12-19T18:47:47.168 INFO:teuthology.orchestra.run.smithi059.stdout:required_client_features {} 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure 0 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:last_failure_osd_epoch 47 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:max_mds 1 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:in 0 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:up {0=14474} 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:failed 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:damaged 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:stopped 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:data_pools [3] 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:metadata_pool 2 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:inline_data enabled 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:balancer 2023-12-19T18:47:47.169 INFO:teuthology.orchestra.run.smithi059.stdout:standby_count_wanted 1 2023-12-19T18:47:47.170 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.rtqzmc{0:14474} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.59:6828/222794355,v1:172.21.15.59:6829/222794355] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:47:47.170 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:47:47.170 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:47:47.170 INFO:teuthology.orchestra.run.smithi059.stdout:Standby daemons: 2023-12-19T18:47:47.170 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-19T18:47:47.170 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi059.abqklt{-1:14486} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.59:6826/1872103080,v1:172.21.15.59:6827/1872103080] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:47:47.170 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi089.dpgsbt{-1:24291} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.89:6824/1622105230,v1:172.21.15.89:6825/1622105230] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:47:47.170 INFO:teuthology.orchestra.run.smithi059.stdout:[mds.cephfs.smithi089.dfrkla{-1:24301} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.89:6826/3691364030,v1:172.21.15.89:6827/3691364030] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fafbc068f60 msgr2=0x7fafbc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fafbc068f60 0x7fafbc06b410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fafcc004890 tx=0x7fafcc009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fcd00 msgr2=0x7fafd010fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fcd00 0x7fafd010fcc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fafc0002a50 tx=0x7fafc00046b0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 shutdown_connections 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fafd00fbaf0 0x7fafd010f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7fafbc068f60 0x7fafbc06b410 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 --2- 172.21.15.59:0/1291635298 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fafd00fcd00 0x7fafd010fcc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 >> 172.21.15.59:0/1291635298 conn(0x7fafd00f7190 msgr2=0x7fafd0065370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:47.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.169+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 shutdown_connections 2023-12-19T18:47:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.170+0000 7fafd76d3700 1 -- 172.21.15.59:0/1291635298 wait complete. 2023-12-19T18:47:47.172 INFO:teuthology.orchestra.run.smithi059.stderr:dumped fsmap epoch 15 2023-12-19T18:47:47.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 -- 172.21.15.59:0/4247601386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fcd00 msgr2=0x7f97000fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:47.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 --2- 172.21.15.59:0/4247601386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fcd00 0x7f97000fd150 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f96ec004940 tx=0x7f96ec009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:47.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 -- 172.21.15.59:0/4247601386 shutdown_connections 2023-12-19T18:47:47.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 --2- 172.21.15.59:0/4247601386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fcd00 0x7f97000fd150 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 --2- 172.21.15.59:0/4247601386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97000fbaf0 0x7f97000fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 -- 172.21.15.59:0/4247601386 >> 172.21.15.59:0/4247601386 conn(0x7f97000f7190 msgr2=0x7f97000f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:47.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 -- 172.21.15.59:0/4247601386 shutdown_connections 2023-12-19T18:47:47.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 -- 172.21.15.59:0/4247601386 wait complete. 2023-12-19T18:47:47.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 Processor -- start 2023-12-19T18:47:47.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.315+0000 7f97071d6700 1 -- start start 2023-12-19T18:47:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f97071d6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fbaf0 0x7f970010c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f97071d6700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97000fcd00 0x7f970010c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:47.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f97071d6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970010cf10 con 0x7f97000fbaf0 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f97071d6700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970010d7b0 con 0x7f97000fcd00 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f9704f72700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fbaf0 0x7f970010c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f9704f72700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fbaf0 0x7f970010c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46056/0 (socket says 172.21.15.59:46056) 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f9704f72700 1 -- 172.21.15.59:0/1722665031 learned_addr learned my addr 172.21.15.59:0/1722665031 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f96fffff700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97000fcd00 0x7f970010c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f9704f72700 1 -- 172.21.15.59:0/1722665031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97000fcd00 msgr2=0x7f970010c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f9704f72700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97000fcd00 0x7f970010c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.316+0000 7f9704f72700 1 -- 172.21.15.59:0/1722665031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec005040 con 0x7f97000fbaf0 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.317+0000 7f9704f72700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fbaf0 0x7f970010c3d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f96f4010d80 tx=0x7f96f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.317+0000 7f96fd7fa700 1 -- 172.21.15.59:0/1722665031 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96f400ed30 con 0x7f97000fbaf0 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.317+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970010da30 con 0x7f97000fbaf0 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.317+0000 7f96fd7fa700 1 -- 172.21.15.59:0/1722665031 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96f400ee90 con 0x7f97000fbaf0 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.317+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970010df70 con 0x7f97000fbaf0 2023-12-19T18:47:47.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.317+0000 7f96fd7fa700 1 -- 172.21.15.59:0/1722665031 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96f4012670 con 0x7f97000fbaf0 2023-12-19T18:47:47.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.318+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e40052e0 con 0x7f97000fbaf0 2023-12-19T18:47:47.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.319+0000 7f96fd7fa700 1 -- 172.21.15.59:0/1722665031 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f96f4018030 con 0x7f97000fbaf0 2023-12-19T18:47:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.319+0000 7f96fd7fa700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96f0068f60 0x7f96f006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.320+0000 7f96fd7fa700 1 -- 172.21.15.59:0/1722665031 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f96f4016070 con 0x7f97000fbaf0 2023-12-19T18:47:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.320+0000 7f96fffff700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96f0068f60 0x7f96f006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:47.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.320+0000 7f96fffff700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96f0068f60 0x7f96f006b410 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f96ec000c00 tx=0x7f96ec009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:47.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.324+0000 7f96fd7fa700 1 -- 172.21.15.59:0/1722665031 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f96f40591a0 con 0x7f97000fbaf0 2023-12-19T18:47:47.355 INFO:tasks.workunit.client.1.smithi089.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2023-12-19T18:47:47.366 INFO:tasks.workunit.client.1.smithi089.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2023-12-19T18:47:47.367 INFO:tasks.workunit.client.1.smithi089.stderr:+ make 2023-12-19T18:47:47.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.591+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 --> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f96e4000bf0 con 0x7f96f0068f60 2023-12-19T18:47:47.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.593+0000 7f96fd7fa700 1 -- 172.21.15.59:0/1722665031 <== mgr.14210 v2:172.21.15.59:6800/3406589371 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+328 (secure 0 0 0) 0x7f96e4000bf0 con 0x7f96f0068f60 2023-12-19T18:47:47.594 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-12-19T18:47:47.594 INFO:teuthology.orchestra.run.smithi059.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5", 2023-12-19T18:47:47.594 INFO:teuthology.orchestra.run.smithi059.stdout: "in_progress": true, 2023-12-19T18:47:47.594 INFO:teuthology.orchestra.run.smithi059.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-19T18:47:47.594 INFO:teuthology.orchestra.run.smithi059.stdout: "services_complete": [], 2023-12-19T18:47:47.594 INFO:teuthology.orchestra.run.smithi059.stdout: "progress": "", 2023-12-19T18:47:47.594 INFO:teuthology.orchestra.run.smithi059.stdout: "message": "Currently upgrading mgr daemons", 2023-12-19T18:47:47.594 INFO:teuthology.orchestra.run.smithi059.stdout: "is_paused": false 2023-12-19T18:47:47.595 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-12-19T18:47:47.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.595+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96f0068f60 msgr2=0x7f96f006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.595+0000 7f97071d6700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96f0068f60 0x7f96f006b410 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f96ec000c00 tx=0x7f96ec009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.595+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fbaf0 msgr2=0x7f970010c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.595+0000 7f97071d6700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fbaf0 0x7f970010c3d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f96f4010d80 tx=0x7f96f400b040 comp rx=0 tx=0).stop 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.595+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 shutdown_connections 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.595+0000 7f97071d6700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f96f0068f60 0x7f96f006b410 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.595+0000 7f97071d6700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97000fbaf0 0x7f970010c3d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.595+0000 7f97071d6700 1 --2- 172.21.15.59:0/1722665031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97000fcd00 0x7f970010c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.595+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 >> 172.21.15.59:0/1722665031 conn(0x7f97000f7190 msgr2=0x7f97000fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.596+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 shutdown_connections 2023-12-19T18:47:47.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.596+0000 7f97071d6700 1 -- 172.21.15.59:0/1722665031 wait complete. 2023-12-19T18:47:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.713+0000 7f7e944b5700 1 -- 172.21.15.59:0/4009150738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c0ffdd0 msgr2=0x7f7e8c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.713+0000 7f7e944b5700 1 --2- 172.21.15.59:0/4009150738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c0ffdd0 0x7f7e8c1021d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c004940 tx=0x7f7e7c009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.713+0000 7f7e944b5700 1 -- 172.21.15.59:0/4009150738 shutdown_connections 2023-12-19T18:47:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.713+0000 7f7e944b5700 1 --2- 172.21.15.59:0/4009150738 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c0ffdd0 0x7f7e8c1021d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.713+0000 7f7e944b5700 1 --2- 172.21.15.59:0/4009150738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e8c066030 0x7f7e8c0ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.713+0000 7f7e944b5700 1 -- 172.21.15.59:0/4009150738 >> 172.21.15.59:0/4009150738 conn(0x7f7e8c0f7190 msgr2=0x7f7e8c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.713+0000 7f7e944b5700 1 -- 172.21.15.59:0/4009150738 shutdown_connections 2023-12-19T18:47:47.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.714+0000 7f7e944b5700 1 -- 172.21.15.59:0/4009150738 wait complete. 2023-12-19T18:47:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.714+0000 7f7e944b5700 1 Processor -- start 2023-12-19T18:47:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.714+0000 7f7e944b5700 1 -- start start 2023-12-19T18:47:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.714+0000 7f7e944b5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c066030 0x7f7e8c10d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.714+0000 7f7e944b5700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e8c0ffdd0 0x7f7e8c10da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.714+0000 7f7e944b5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c10e020 con 0x7f7e8c066030 2023-12-19T18:47:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.714+0000 7f7e944b5700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c10e8c0 con 0x7f7e8c0ffdd0 2023-12-19T18:47:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.714+0000 7f7e92251700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c066030 0x7f7e8c10d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:47.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e92251700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c066030 0x7f7e8c10d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:46066/0 (socket says 172.21.15.59:46066) 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e92251700 1 -- 172.21.15.59:0/1676895708 learned_addr learned my addr 172.21.15.59:0/1676895708 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e91a50700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e8c0ffdd0 0x7f7e8c10da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e92251700 1 -- 172.21.15.59:0/1676895708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e8c0ffdd0 msgr2=0x7f7e8c10da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e92251700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e8c0ffdd0 0x7f7e8c10da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e92251700 1 -- 172.21.15.59:0/1676895708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c005040 con 0x7f7e8c066030 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e92251700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c066030 0x7f7e8c10d4e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7e88002a80 tx=0x7f7e880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e82ffd700 1 -- 172.21.15.59:0/1676895708 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e88007cc0 con 0x7f7e8c066030 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8c10eba0 con 0x7f7e8c066030 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8c10f0e0 con 0x7f7e8c066030 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e82ffd700 1 -- 172.21.15.59:0/1676895708 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e8800f040 con 0x7f7e8c066030 2023-12-19T18:47:47.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.715+0000 7f7e82ffd700 1 -- 172.21.15.59:0/1676895708 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e8800da60 con 0x7f7e8c066030 2023-12-19T18:47:47.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.717+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c04bb80 con 0x7f7e8c066030 2023-12-19T18:47:47.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.717+0000 7f7e82ffd700 1 -- 172.21.15.59:0/1676895708 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7e8800dc60 con 0x7f7e8c066030 2023-12-19T18:47:47.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.718+0000 7f7e82ffd700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e78068f60 0x7f7e7806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T18:47:47.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.718+0000 7f7e82ffd700 1 -- 172.21.15.59:0/1676895708 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7e88086ad0 con 0x7f7e8c066030 2023-12-19T18:47:47.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.718+0000 7f7e91a50700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e78068f60 0x7f7e7806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T18:47:47.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.718+0000 7f7e91a50700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e78068f60 0x7f7e7806b410 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c004890 tx=0x7f7e7c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T18:47:47.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:47.723+0000 7f7e82ffd700 1 -- 172.21.15.59:0/1676895708 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7e880569e0 con 0x7f7e8c066030 2023-12-19T18:47:47.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:47 smithi089 ceph-mon[88459]: from='client.24423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:47.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:47 smithi089 ceph-mon[88459]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:47.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:47 smithi089 ceph-mon[88459]: pgmap v234: 65 pgs: 65 active+clean; 169 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 7.3 MiB/s wr, 833 op/s 2023-12-19T18:47:47.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:47 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/394964430' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-19T18:47:47.755 INFO:journalctl@ceph.mon.smithi089.smithi089.stdout:Dec 19 18:47:47 smithi089 ceph-mon[88459]: from='client.? 172.21.15.59:0/1291635298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:47:47.816 INFO:tasks.workunit.client.1.smithi089.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2023-12-19T18:47:47.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:47 smithi059 ceph-mon[80661]: from='client.24423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:47.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:47 smithi059 ceph-mon[80661]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T18:47:47.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:47 smithi059 ceph-mon[80661]: pgmap v234: 65 pgs: 65 active+clean; 169 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 7.3 MiB/s wr, 833 op/s 2023-12-19T18:47:47.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:47 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/394964430' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-19T18:47:47.949 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:47:47 smithi059 ceph-mon[80661]: from='client.? 172.21.15.59:0/1291635298' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T18:47:48.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.012+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7e8c10ed30 con 0x7f7e8c066030 2023-12-19T18:47:48.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.012+0000 7f7e82ffd700 1 -- 172.21.15.59:0/1676895708 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f7e8c10ed30 con 0x7f7e8c066030 2023-12-19T18:47:48.014 INFO:teuthology.orchestra.run.smithi059.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2023-12-19T18:47:48.014 INFO:teuthology.orchestra.run.smithi059.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2023-12-19T18:47:48.014 INFO:teuthology.orchestra.run.smithi059.stdout: fs cephfs has deprecated feature inline_data enabled. 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e78068f60 msgr2=0x7f7e7806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e78068f60 0x7f7e7806b410 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c004890 tx=0x7f7e7c009e30 comp rx=0 tx=0).stop 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c066030 msgr2=0x7f7e8c10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c066030 0x7f7e8c10d4e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7e88002a80 tx=0x7f7e880046b0 comp rx=0 tx=0).stop 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 shutdown_connections 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:6800/3406589371,v1:172.21.15.59:6801/3406589371] conn(0x7f7e78068f60 0x7f7e7806b410 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7e8c066030 0x7f7e8c10d4e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 --2- 172.21.15.59:0/1676895708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7e8c0ffdd0 0x7f7e8c10da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 >> 172.21.15.59:0/1676895708 conn(0x7f7e8c0f7190 msgr2=0x7f7e8c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T18:47:48.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 shutdown_connections 2023-12-19T18:47:48.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:47:48.015+0000 7f7e944b5700 1 -- 172.21.15.59:0/1676895708 wait complete. 2023-12-19T18:48:09.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:09.598+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:09.598+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:09.598+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:10.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:10.549+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:10.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:10.549+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:10.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:10.549+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:11.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:11.576+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:11.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:11.576+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:11.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:11.576+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:12.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:12.604+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:12.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:12.604+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:12.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:12.604+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:13.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:13.577+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:13.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:13.577+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:13.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:13.577+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:14.311 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:13.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:14.312 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:13.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:14.312 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:13.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:14.310+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:14.310+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:14.310+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:14.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:14.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:14.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:14.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:14.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:14.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:15.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:14.979+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:15.342 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:14.979+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:15.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:14.979+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:15.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:15.341+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:15.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:15.341+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:15.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:15.341+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:15.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:15.589+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:15.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:15.589+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:15.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:15.589+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:16.346 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:15.993+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:16.346 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:15.993+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:16.346 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:15.993+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:16.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:16.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:16.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:16.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:16.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:16.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:16.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:16.631+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:16.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:16.631+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:16.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:16.631+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:17.356 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:17.034+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:17.356 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:17.034+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:17.356 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:17.034+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:17.662 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:17.355+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:17.662 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:17.355+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:17.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:17.355+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:17.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:17.661+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:17.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:17.661+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:17.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:17.661+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:18.375 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:18.042+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:18.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:18.042+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:18.376 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:18.042+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:18.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:18.374+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:18.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:18.374+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:18.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:18.374+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:18.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:18.626+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:18.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:18.626+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:18.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:18.626+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:19.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:19.009+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:19.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:19.009+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:19.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:19.009+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:19.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:19.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:19.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:19.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:19.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:19.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:19.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:19.605+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:19.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:19.605+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:19.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:19.605+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:20.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:19.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:20.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:19.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:20.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:19.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:20.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:20.403+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:20.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:20.403+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:20.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:20.403+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:20.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:20.615+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:20.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:20.615+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:20.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:20.615+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:21.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:20.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:21.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:20.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:21.364 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:20.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:21.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:21.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:21.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:21.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:21.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:21.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:21.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:21.581+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:21.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:21.581+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:21.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:21.581+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:22.392 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:22.006+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:22.392 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:22.006+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:22.392 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:22.006+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:22.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:22.390+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:22.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:22.390+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:22.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:22.390+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:22.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:22.618+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:22.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:22.618+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:22.618+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:23.351 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:23.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:23.352 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:23.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:23.352 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:23.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:23.629 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:23.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:23.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:23.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:23.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:23.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:23.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:23.628+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:23.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:23.628+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:23.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:23.628+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:24.358 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:24.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:24.358 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:24.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:24.358 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:24.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:24.656 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:24.356+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:24.656 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:24.356+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:24.656 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:24.356+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:24.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:24.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:24.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:24.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:24.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:24.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:25.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:25.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:25.339 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:25.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:25.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:25.337+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:25.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:25.337+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:25.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:25.337+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:25.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:25.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:25.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:25.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:25.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:25.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:26.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:26.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:26.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:26.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:26.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:26.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:26.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:26.358+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:26.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:26.358+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:26.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:26.358+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:26.987 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:26.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:26.987 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:26.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:26.987 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:26.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:27.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:27.104+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:27.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:27.104+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:27.399 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:27.104+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:27.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:27.397+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:27.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:27.397+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:27.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:27.397+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:28.066 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:27.747+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:28.066 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:27.747+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:28.067 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:27.747+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:28.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:28.065+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:28.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:28.065+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:28.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:28.065+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:28.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:28.445+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:28.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:28.445+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:28.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:28.445+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:29.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:28.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:29.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:28.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:29.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:28.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:29.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:29.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:29.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:29.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:29.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:29.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:29.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:29.460+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:29.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:29.460+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:29.712 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:29.460+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:30.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:29.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:30.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:29.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:30.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:29.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:30.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:30.089+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:30.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:30.089+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:30.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:30.089+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:30.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:30.431+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:30.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:30.431+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:30.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:30.431+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:30.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:30.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:30.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:30.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:30.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:30.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:31.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:31.127+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:31.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:31.127+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:31.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:31.127+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:31.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:31.479+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:31.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:31.479+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:31.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:31.479+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:31.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:31.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:31.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:31.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:31.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:31.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:32.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:32.161+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:32.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:32.161+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:32.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:32.161+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:32.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:32.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:32.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:32.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:32.692 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:32.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:32.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:32.691+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:32.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:32.691+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:32.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:32.691+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:33.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:33.131+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:33.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:33.131+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:33.435 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:33.131+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:33.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:33.433+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:33.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:33.433+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:33.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:33.433+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:33.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:33.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:33.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:33.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:33.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:33.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:34.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:34.158+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:34.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:34.158+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:34.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:34.158+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:34.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:34.456+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:34.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:34.456+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:34.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:34.456+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:34.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:34.622+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:34.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:34.622+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:34.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:34.622+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:35.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:35.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:35.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:35.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:35.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:35.455+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:35.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:35.455+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:35.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:35.455+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:35.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:35.652+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:35.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:35.652+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:35.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:35.652+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:36.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:36.156+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:36.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:36.156+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:36.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:36.156+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:36.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:36.421+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:36.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:36.421+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:36.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:36.421+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:36.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:36.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:36.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:36.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:36.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:36.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:37.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:37.187+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:37.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:37.187+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:37.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:37.187+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:37.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:37.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:37.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:37.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:37.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:37.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:37.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:37.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:37.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:37.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:37.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:37.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:38.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:38.208+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:38.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:38.208+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:38.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:38.208+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:38.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:38.394+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:38.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:38.394+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:38.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:38.394+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:38.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:38.689+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:38.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:38.689+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:38.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:38.689+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:39.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:39.428+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:39.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:39.428+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:39.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:39.428+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:39.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:39.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:39.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:39.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:40.139 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:39.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:40.139 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:39.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:40.139 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:39.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:40.407 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:40.138+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:40.408 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:40.138+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:40.408 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:40.138+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:40.678 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:40.406+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:40.678 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:40.406+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:40.678 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:40.406+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:40.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:40.676+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:40.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:40.677+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:40.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:40.677+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:41.382 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:41.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:41.382 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:41.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:41.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:41.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:41.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:41.381+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:41.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:41.381+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:41.691 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:41.381+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:41.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:41.690+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:41.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:41.690+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:41.690+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:42.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:42.107+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:42.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:42.107+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:42.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:42.107+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:42.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:42.356+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:42.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:42.356+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:42.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:42.356+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:43.123 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:42.704+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:43.123 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:42.704+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:43.123 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:42.704+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:43.370+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:43.370+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:43.370+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:43.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:43.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:43.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:43.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:43.738+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:43.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:43.738+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:43.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:43.738+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:44.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:44.357+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:44.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:44.357+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:44.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:44.357+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:44.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:44.168+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:44.168+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:44.168+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:45.181 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:44.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:45.181 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:44.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:45.181 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:44.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:45.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:45.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:45.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:45.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:45.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:45.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:45.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:45.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:45.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:45.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:45.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:45.686+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:45.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:45.686+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:45.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:45.686+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:46.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:46.225+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:46.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:46.225+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:46.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:46.225+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:46.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:46.365+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:46.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:46.365+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:46.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:46.365+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:46.977 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:46.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:46.977 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:46.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:46.977 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:46.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:47.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:47.261+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:47.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:47.261+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:47.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:47.261+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:47.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:47.395+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:47.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:47.395+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:47.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:47.395+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:47.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:47.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:47.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:47.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:47.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:47.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:48.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:48.371+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:48.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:48.371+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:48.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:48.371+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:48.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:48.243+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:48.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:48.243+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:48.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:48.243+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:49.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:48.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:49.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:48.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:49.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:48.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:49.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:49.195+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:49.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:49.195+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:49.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:49.195+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:49.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:49.405+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:49.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:49.405+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:49.405+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:49.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:49.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:49.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:49.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:49.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:49.665+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:50.639 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:50.209+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:50.639 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:50.209+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:50.639 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:50.209+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:50.639 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:50.435+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:50.639 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:50.435+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:50.639 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:50.435+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:50.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:50.637+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:50.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:50.637+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:50.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:50.637+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:51.594 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:51.406+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:51.594 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:51.406+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:51.594 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:51.406+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:51.594 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:51.243+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:51.594 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:51.243+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:51.594 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:51.243+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:51.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:51.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:51.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:51.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:51.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:51.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:52.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:52.238+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:52.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:52.238+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:52.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:52.238+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:52.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:52.371+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:52.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:52.371+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:52.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:52.371+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:52.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:52.625+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:52.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:52.625+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:52.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:52.625+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:53.517 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:53.376+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:53.517 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:53.376+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:53.517 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:53.376+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:53.517 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:53.251+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:53.517 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:53.251+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:53.517 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:53.251+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:53.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:53.594+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:53.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:53.594+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:53.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:53.594+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:54.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:54.218+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:54.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:54.218+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:54.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:54.218+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:54.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:54.399+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:54.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:54.399+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:54.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:54.399+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:54.625+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:54.625+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:54.625+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:55.671 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:55.213+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:55.671 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:55.213+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:55.671 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:55.213+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:55.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:55.388+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:55.672 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:55.388+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:55.672 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:55.388+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:55.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:55.670+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:55.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:55.670+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:55.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:55.670+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:56.660 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:56.421+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:56.660 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:56.421+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:56.660 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:56.421+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:56.660 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:56.205+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:56.661 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:56.205+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:56.661 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:56.205+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:56.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:56.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:56.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:56.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:56.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:56.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:57.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:57.399+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:57.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:57.399+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:57.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:57.399+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:57.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:57.248+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:57.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:57.248+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:57.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:57.248+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:58.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:57.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:58.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:57.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:58.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:57.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:58.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:58.437+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:58.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:58.437+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:58.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:58.437+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:58.687 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:58.213+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:58.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:58.213+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:58.688 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:58.213+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:58.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:58.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:58.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:58.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:58.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:58.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:59.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:59.173+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:59.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:59.173+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:59.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:48:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:48:59.173+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:48:59.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:59.460+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:59.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:59.460+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:59.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:48:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:48:59.460+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:48:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:59.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:59.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:48:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:48:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:48:59.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:00.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:00.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:00.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:00.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:00.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:00.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:00.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:00.510+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:00.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:00.510+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:00.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:00.510+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:00.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:00.659+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:00.659+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:00.659+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:01.499 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:01.227+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:01.499 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:01.227+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:01.499 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:01.227+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:01.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:01.498+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:01.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:01.498+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:01.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:01.498+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:01.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:01.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:01.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:02.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:02.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:02.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:02.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:02.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:02.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:02.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:02.514+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:02.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:02.514+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:02.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:02.514+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:02.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:02.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:02.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:03.517 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:03.211+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:03.517 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:03.211+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:03.517 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:03.211+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:03.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:03.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:03.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:03.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:03.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:03.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:03.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:04.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:04.196+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:04.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:04.196+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:04.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:04.196+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:04.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:04.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:04.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:04.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:04.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:04.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:04.564+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:04.564+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:04.564+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:05.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:05.209+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:05.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:05.209+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:05.570 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:05.209+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:05.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:05.568+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:05.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:05.568+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:05.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:05.568+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:05.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:05.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:05.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:05.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:06.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:06.175+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:06.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:06.175+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:06.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:06.175+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:06.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:06.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:06.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:06.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:06.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:06.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:06.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:06.671+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:06.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:06.671+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:06.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:06.671+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:07.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:07.190+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:07.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:07.190+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:07.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:07.190+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:07.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:07.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:07.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:07.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:07.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:07.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:07.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:07.530+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:07.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:07.530+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:07.530+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:08.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:08.182+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:08.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:08.182+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:08.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:08.182+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:08.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:08.644+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:08.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:08.644+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:08.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:08.644+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:08.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:08.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:08.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:08.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:09.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:09.159+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:09.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:09.159+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:09.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:09.159+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:09.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:09.671+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:09.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:09.671+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:09.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:09.671+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:09.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:09.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:09.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:09.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:09.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:09.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:10.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:10.123+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:10.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:10.123+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:10.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:10.123+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:10.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:10.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:10.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:10.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:10.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:10.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:10.651+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:10.651+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:10.651+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:11.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:11.128+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:11.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:11.128+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:11.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:11.128+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:11.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:11.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:11.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:11.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:11.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:11.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:11.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:11.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:11.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:11.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:11.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:11.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:12.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:12.086+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:12.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:12.086+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:12.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:12.086+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:12.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:12.496+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:12.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:12.496+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:12.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:12.496+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:12.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:12.653+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:12.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:12.653+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:12.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:12.653+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:13.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:13.041+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:13.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:13.041+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:13.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:13.041+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:13.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:13.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:13.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:13.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:13.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:13.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:13.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:13.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:13.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:13.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:13.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:13.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:14.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:14.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:14.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:14.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:14.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:14.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:14.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:14.587+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:14.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:14.587+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:14.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:14.587+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:14.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:14.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:14.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:15.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:15.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:15.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:15.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:15.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:15.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:15.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:15.635+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:15.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:15.635+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:15.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:15.635+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:15.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:15.561+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:15.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:15.561+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:15.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:15.561+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:16.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:16.026+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:16.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:16.026+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:16.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:16.026+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:16.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:16.567+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:16.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:16.567+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:16.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:16.567+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:16.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:16.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:16.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:17.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:17.064+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:17.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:17.064+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:17.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:17.064+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:17.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:17.637+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:17.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:17.637+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:17.637+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:17.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:17.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:17.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:18.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:18.023+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:18.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:18.023+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:18.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:18.023+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:18.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:18.641+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:18.641+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:18.641+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:18.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:18.563+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:18.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:18.563+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:18.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:18.563+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:19.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:19.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:19.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:19.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:19.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:19.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:19.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:19.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:19.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:19.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:19.584+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:19.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:19.687+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:19.687+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:19.687+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:20.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:20.087+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:20.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:20.087+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:20.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:20.087+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:20.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:20.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:20.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:20.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:20.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:20.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:20.724+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:20.724+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:20.724+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:21.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:21.106+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:21.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:21.106+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:21.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:21.106+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:21.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:21.588+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:21.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:21.588+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:21.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:21.588+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:21.771+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:21.771+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:21.771+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:22.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:22.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:22.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:22.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:22.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:22.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:22.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:22.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:22.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:22.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:22.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:22.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:22.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:22.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:22.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:22.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:22.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:23.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:23.047+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:23.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:23.047+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:23.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:23.047+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:23.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:23.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:23.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:23.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:23.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:23.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:24.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:23.815+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:24.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:23.815+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:24.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:23.815+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:24.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:24.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:24.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:24.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:24.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:24.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:24.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:24.571+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:24.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:24.571+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:24.828 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:24.571+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:25.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:25.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:25.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:25.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:25.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:25.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:24.826+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:24.826+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:24.826+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:25.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:25.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:25.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:25.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:25.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:25.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:25.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:25.807+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:25.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:25.807+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:25.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:25.807+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:26.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:26.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:26.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:26.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:26.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:26.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:26.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:26.606+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:26.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:26.606+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:26.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:26.606+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:26.788+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:26.788+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:26.788+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:27.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:27.030+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:27.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:27.030+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:27.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:27.030+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:27.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:27.740+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:27.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:27.740+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:27.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:27.740+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:27.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:27.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:27.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:28.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:28.076+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:28.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:28.076+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:28.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:28.076+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:28.781 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:28.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:28.782 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:28.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:28.782 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:28.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:29.047 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:28.780+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:29.047 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:28.780+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:29.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:28.780+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:29.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:29.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:29.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:29.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:29.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:29.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:29.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:29.664+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:29.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:29.664+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:29.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:29.664+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:29.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:29.754+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:29.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:29.754+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:29.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:29.754+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:30.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:30.031+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:30.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:30.031+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:30.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:30.031+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:30.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:30.648+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:30.648+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:30.648+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:30.749+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:30.749+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:30.749+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:31.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:31.012+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:31.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:31.012+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:31.374 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:31.012+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:31.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:31.691+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:31.691+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:31.691+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:31.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:31.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:31.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:32.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:32.013+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:32.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:32.013+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:32.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:32.013+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:32.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:32.710+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:32.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:32.710+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:32.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:32.710+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:32.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:32.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:32.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:32.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:32.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:32.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:33.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:33.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:33.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:33.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:33.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:33.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:33.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:33.615+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:33.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:33.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:33.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:33.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:33.744+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:33.744+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:33.744+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:34.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:34.072+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:34.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:34.072+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:34.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:34.072+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:34.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:34.664+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:34.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:34.664+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:34.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:34.664+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:35.025 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:34.765+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:35.025 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:34.765+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:35.025 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:34.765+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:35.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:35.023+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:35.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:35.023+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:35.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:35.023+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:35.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:35.753+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:35.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:35.753+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:35.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:35.753+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:35.675+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:35.675+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:35.675+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:36.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:36.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:36.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:36.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:36.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:36.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:36.982 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:36.798+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:36.982 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:36.798+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:36.982 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:36.798+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:36.983 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:36.703+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:36.983 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:36.703+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:36.983 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:36.703+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:37.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:37.055+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:37.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:37.055+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:37.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:37.055+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:37.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:37.675+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:37.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:37.675+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:37.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:37.675+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:38.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:38.008+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:38.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:38.008+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:38.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:38.008+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:37.822+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:37.822+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:37.822+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:38.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:38.679+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:38.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:38.679+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:38.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:38.679+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:38.817+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:38.817+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:38.817+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:39.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:39.020+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:39.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:39.020+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:39.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:39.020+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:39.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:39.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:39.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:39.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:39.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:39.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:39.774+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:39.774+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:39.774+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:40.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:40.069+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:40.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:40.069+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:40.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:40.069+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:40.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:40.671+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:40.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:40.671+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:40.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:40.671+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:40.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:40.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:40.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:40.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:40.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:40.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:41.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:41.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:41.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:41.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:41.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:41.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:41.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:41.638+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:41.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:41.638+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:41.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:41.638+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:41.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:41.849+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:41.849+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:41.849+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:42.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:42.100+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:42.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:42.100+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:42.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:42.100+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:42.889 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:42.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:42.889 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:42.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:42.890 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:42.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:43.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:43.078+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:43.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:43.078+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:43.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:43.078+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:43.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:42.888+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:43.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:42.888+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:43.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:42.888+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:43.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:43.647+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:43.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:43.647+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:43.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:43.647+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:44.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:44.064+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:44.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:44.064+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:44.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:44.064+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:44.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:43.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:44.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:43.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:44.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:43.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:44.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:44.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:44.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:44.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:44.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:44.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:45.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:45.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:45.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:45.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:45.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:45.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:45.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:44.905+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:44.905+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:44.905+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:45.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:45.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:45.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:45.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:45.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:45.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:45.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:45.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:45.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:45.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:45.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:45.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:46.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:46.084+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:46.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:46.084+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:46.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:46.084+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:46.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:46.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:46.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:46.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:46.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:46.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:47.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:46.877+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:47.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:46.877+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:47.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:46.877+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:47.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:47.130+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:47.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:47.130+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:47.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:47.130+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:47.887 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:47.565+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:47.887 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:47.565+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:47.887 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:47.565+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:48.155 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:47.884+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:48.156 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:47.884+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:48.156 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:47.884+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:48.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:48.154+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:48.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:48.154+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:48.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:48.154+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:48.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:48.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:48.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:48.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:48.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:48.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:49.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:48.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:49.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:48.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:49.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:48.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:49.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:49.150+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:49.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:49.150+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:49.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:49.150+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:49.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:49.549+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:49.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:49.549+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:49.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:49.549+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:50.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:50.111+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:50.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:50.111+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:50.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:50.111+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:50.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:49.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:50.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:49.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:50.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:49.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:50.858 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:50.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:50.858 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:50.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:50.858 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:50.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:51.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:51.103+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:51.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:51.103+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:51.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:51.103+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:51.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:50.856+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:51.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:50.856+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:51.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:50.856+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:51.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:51.848+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:51.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:51.848+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:51.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:51.848+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:51.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:51.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:51.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:52.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:49:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:49:51.973+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:49:52.520 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:52.129+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:52.520 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:52.129+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:52.520 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:52.129+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:52.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:52.668+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:52.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:52.668+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:52.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:52.668+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:52.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:52.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:52.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:53.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:53.097+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:53.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:53.097+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:53.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:53.097+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:53.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:53.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:53.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:53.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:53.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:53.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:54.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:54.112+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:54.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:54.112+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:54.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:54.112+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:54.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:53.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:54.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:53.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:54.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:53.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:54.863+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:54.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:54.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:54.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:54.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:54.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:54.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:54.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:54.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:55.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:55.084+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:55.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:55.084+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:55.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:55.084+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:55.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:55.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:55.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:55.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:55.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:55.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:56.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:56.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:56.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:56.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:56.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:56.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:55.904+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:55.904+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:55.904+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:56.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:56.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:56.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:56.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:56.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:56.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:56.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:57.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:49:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:49:56.974+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:49:57.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:57.074+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:57.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:57.074+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:57.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:57.074+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:57.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:57.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:57.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:57.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:57.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:57.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:58.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:58.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:58.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:58.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:58.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:58.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:59.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:58.715+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:59.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:58.715+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:59.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:58.715+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:59.112 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:58.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:59.112 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:58.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:59.112 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:58.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:59.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:59.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:59.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:59.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:59.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:49:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:49:59.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:49:59.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:59.666+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:59.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:59.666+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:59.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:49:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:49:59.666+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:49:59.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:59.866+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:59.866+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:49:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:49:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:49:59.866+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:00.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:00.153+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:00.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:00.153+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:00.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:00.153+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:00.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:00.620+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:00.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:00.620+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:00.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:00.620+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:01.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:00.910+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:01.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:00.910+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:01.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:00.910+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:01.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:01.134+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:01.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:01.134+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:01.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:01.134+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:01.932 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:01.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:01.932 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:01.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:01.932 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:01.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:02.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:01.974+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:02.092+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:02.092+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:02.092+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:02.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:01.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:02.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:01.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:02.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:01.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:02.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:02.633+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:02.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:02.633+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:02.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:02.633+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:03.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:03.104+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:03.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:03.104+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:03.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:03.104+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:03.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:02.929+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:03.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:02.929+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:03.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:02.929+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:03.611+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:03.611+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:03.611+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:04.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:04.109+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:04.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:04.109+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:04.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:04.109+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:03.978+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:03.978+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:03.978+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:04.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:04.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:04.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:05.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:05.027+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:05.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:05.027+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:05.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:05.027+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:05.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:05.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:05.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:05.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:05.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:05.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:05.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:05.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:06.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:06.022+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:06.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:06.022+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:06.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:06.022+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:06.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:06.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:06.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:06.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:06.671+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:06.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:06.671+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:06.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:06.671+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:07.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:06.975+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 38 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:07.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:07.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:07.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:07.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:07.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:07.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:07.014+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:07.014+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:07.014+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:07.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:07.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:07.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:07.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:08.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:08.121+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:08.121+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:08.121+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:08.023+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:08.023+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:08.023+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:09.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:08.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:09.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:08.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:09.058 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:08.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:09.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:09.162+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:09.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:09.162+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:09.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:09.162+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:09.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:09.056+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:09.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:09.056+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:09.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:09.056+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:10.093 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:09.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:10.093 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:09.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:10.093 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:09.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:10.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:10.091+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:10.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:10.091+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:10.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:10.091+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:10.175+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:10.175+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:10.175+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:11.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:10.716+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:11.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:10.716+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:11.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:10.716+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:11.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:11.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:11.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:11.072+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:11.072+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:11.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:11.072+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:11.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:11.686+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:11.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:11.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:11.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:11.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:12.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:11.975+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 57 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:12.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:12.140+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:12.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:12.140+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:12.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:12.140+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:12.090+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:12.090+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:12.090+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:12.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:12.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:12.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:12.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:12.975 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:12.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:13.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:13.131+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:13.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:13.131+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:13.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:13.131+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:13.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:13.110+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:13.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:13.110+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:13.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:13.110+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:13.651+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:13.651+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:13.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:13.651+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:14.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:14.152+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:14.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:14.152+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:14.152+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:14.145+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:14.145+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:14.145+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:14.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:14.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:14.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:15.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:15.161+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:15.161+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:15.161+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:15.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:15.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:15.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:16.180 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:15.709+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:16.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:15.709+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:16.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:15.709+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:16.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:16.187+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:16.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:16.187+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:16.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:16.187+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:16.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:16.179+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:16.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:16.179+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:16.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:16.179+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:16.977 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:16.702+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:16.978 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:16.702+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:16.978 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:16.702+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:17.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:16.975+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 91 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:17.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:17.218+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:17.218+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:17.218+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:17.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:17.188+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:17.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:17.188+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:17.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:17.188+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:18.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:17.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:18.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:17.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:18.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:17.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:18.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:18.203+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:18.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:18.203+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:18.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:18.203+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:18.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:18.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:18.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:18.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:18.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:18.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:19.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:18.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:19.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:18.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:19.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:18.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:19.683 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:19.208+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:19.683 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:19.208+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:19.684 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:19.208+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:19.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:19.253+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:19.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:19.253+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:19.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:19.253+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:19.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:19.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:19.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:19.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:19.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:19.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:20.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:20.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:20.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:20.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:20.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:20.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:20.222+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:20.222+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:20.222+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:20.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:20.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:20.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:20.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:20.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:20.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:21.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:21.254+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:21.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:21.254+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:21.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:21.254+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:21.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:21.188+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:21.188+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:21.188+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:21.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:21.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:21.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:21.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:22.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:21.976+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 118 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:22.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:22.214+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:22.214+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:22.214+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:22.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:22.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:22.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:22.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:22.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:22.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:23.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:22.713+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:23.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:22.713+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:23.172 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:22.713+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:23.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:23.170+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:23.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:23.170+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:23.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:23.170+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:23.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:23.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:23.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:23.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:23.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:23.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:23.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:23.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:23.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:23.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:24.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:24.202+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:24.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:24.202+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:24.667 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:24.202+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:24.667 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:24.229+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:24.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:24.229+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:24.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:24.229+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:24.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:24.665+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:24.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:24.665+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:24.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:24.665+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:25.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:25.239+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:25.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:25.239+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:25.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:25.239+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:25.217+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:25.217+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:25.217+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:26.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:25.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:26.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:25.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:26.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:25.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:26.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:26.235+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:26.235+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:26.235+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:26.194+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:26.194+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:26.194+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:26.990 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:26.976+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 154 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:26.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:26.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:26.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:26.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:26.991 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:26.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:27.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:27.204+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:27.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:27.204+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:27.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:27.204+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:27.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:27.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:27.697 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:27.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:27.697 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:27.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:27.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:27.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:27.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:27.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:27.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:27.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:28.647 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:28.244+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:28.647 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:28.244+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:28.647 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:28.244+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:28.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:28.254+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:28.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:28.254+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:28.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:28.254+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:28.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:28.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:28.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:28.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:28.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:28.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:29.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:29.276+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:29.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:29.276+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:29.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:29.276+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:29.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:29.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:29.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:29.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:29.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:29.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:29.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:29.673+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:29.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:29.673+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:29.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:29.673+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:30.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:30.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:30.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:30.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:30.676 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:30.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:30.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:30.273+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:30.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:30.273+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:30.677 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:30.273+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:30.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:30.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:30.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:30.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:30.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:30.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:31.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:31.207+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:31.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:31.207+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:31.630 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:31.207+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:31.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:31.232+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:31.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:31.232+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:31.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:31.232+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:31.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:31.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:31.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:31.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:31.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:31.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:32.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:31.977+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 181 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:32.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:32.225+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:32.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:32.225+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:32.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:32.225+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:32.193+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:32.193+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:32.193+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:32.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:32.678+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:32.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:32.679+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:32.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:32.679+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:33.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:33.176+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:33.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:33.176+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:33.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:33.176+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:33.524 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:33.245+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:33.524 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:33.245+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:33.525 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:33.245+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:33.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:33.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:33.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:33.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:33.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:33.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:34.266+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:34.266+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:34.266+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:34.152+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:34.152+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:34.152+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:34.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:34.667+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:34.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:34.667+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:34.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:34.667+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:35.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:35.229+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:35.229+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:35.229+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:35.165+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:35.165+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:35.165+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:35.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:35.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:35.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:35.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:35.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:35.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:36.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:36.120+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:36.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:36.120+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:36.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:36.120+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:36.209+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:36.209+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:36.209+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:36.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:36.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:36.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:36.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:36.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:36.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:37.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:36.977+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 212 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:37.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:37.214+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:37.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:37.214+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:37.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:37.214+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:37.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:37.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:37.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:37.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:37.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:37.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:37.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:37.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:37.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:37.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:38.640 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:38.239+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:38.641 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:38.239+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:38.641 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:38.239+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:38.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:38.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:38.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:38.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:38.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:38.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:38.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:38.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:38.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:38.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:38.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:38.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:39.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:39.185+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:39.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:39.185+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:39.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:39.185+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:39.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:39.250+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:39.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:39.250+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:39.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:39.250+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:39.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:39.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:39.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:39.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:39.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:40.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:40.242+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:40.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:40.242+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:40.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:40.242+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:40.177+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:40.177+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:40.177+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:40.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:40.608+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:40.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:40.608+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:40.608+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:41.187+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:41.187+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:41.187+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:41.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:41.229+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:41.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:41.229+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:41.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:41.229+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:41.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:41.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:41.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:41.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:41.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:42.258 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:41.978+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 239 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:42.258 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:42.196+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:42.258 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:42.196+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:42.258 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:42.196+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:42.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:42.256+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:42.584 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:42.256+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:42.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:42.256+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:42.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:42.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:42.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:42.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:42.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:42.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:43.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:43.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:43.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:43.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:43.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:43.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:43.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:43.288+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:43.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:43.288+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:43.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:43.288+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:43.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:43.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:43.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:43.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:43.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:43.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:44.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:44.306+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:44.306+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:44.306+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:44.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:44.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:44.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:44.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:44.562+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:44.562+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:44.562+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:45.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:45.181+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:45.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:45.181+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:45.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:45.181+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:45.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:45.279+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:45.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:45.279+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:45.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:45.279+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:45.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:45.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:45.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:45.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:45.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:45.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:46.579 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:46.217+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:46.579 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:46.217+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:46.579 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:46.217+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:46.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:46.298+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:46.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:46.298+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:46.579 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:46.298+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:46.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:46.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:46.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:46.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:46.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:46.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:47.318 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:46.979+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 270 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:47.318 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:47.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:47.318 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:47.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:47.318 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:47.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:47.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:47.316+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:47.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:47.316+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:47.585 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:47.316+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:47.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:47.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:47.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:47.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:47.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:47.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:48.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:48.319+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:48.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:48.319+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:48.319+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:48.147+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:48.147+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:48.147+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:48.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:48.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:48.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:48.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:49.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:49.348+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:49.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:49.348+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:49.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:49.348+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:49.110+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:49.110+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:49.110+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:49.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:49.563+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:49.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:49.563+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:49.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:49.563+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:50.394 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:50.115+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:50.394 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:50.115+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:50.394 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:50.115+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:50.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:50.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:50.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:50.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:50.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:50.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:50.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:50.392+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:50.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:50.392+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:50.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:50.392+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:51.405 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:51.145+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:51.405 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:51.145+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:51.405 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:51.145+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:51.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:51.401+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:51.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:51.401+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:51.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:51.401+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:51.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:51.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:51.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:51.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:51.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:51.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:52.361 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:51.979+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 300 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:52.361 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:52.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:52.361 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:52.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:52.361 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:52.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:52.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:52.566+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:52.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:52.566+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:52.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:52.566+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:52.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:52.359+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:52.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:52.359+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:52.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:52.359+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:53.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:53.351+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:53.351+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:53.351+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:53.110+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:53.110+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:53.110+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:53.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:53.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:53.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:53.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:53.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:53.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:54.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:54.325+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:54.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:54.325+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:54.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:54.325+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:54.091+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:54.091+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:54.091+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:54.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:54.565+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:54.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:54.565+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:54.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:54.565+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:55.344 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:55.042+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:55.344 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:55.042+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:55.344 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:55.042+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:55.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:55.556+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:55.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:55.556+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:55.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:55.556+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:55.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:55.342+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:55.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:55.342+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:55.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:55.342+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:56.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:56.050+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:56.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:56.051+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:56.303 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:56.051+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:56.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:56.509+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:56.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:56.509+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:56.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:56.509+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:56.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:56.301+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:56.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:56.301+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:56.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:56.301+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:57.261 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:50:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:50:56.980+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 331 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:50:57.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:57.069+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:57.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:57.069+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:57.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:57.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:57.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:57.260+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:57.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:57.260+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:57.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:57.260+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:57.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:57.498+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:57.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:57.498+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:57.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:57.498+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:58.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:58.264+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:58.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:58.264+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:58.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:58.264+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:58.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:58.045+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:58.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:58.045+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:58.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:58.045+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:58.457+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:58.457+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:58.457+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:59.308 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:59.017+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:59.308 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:59.017+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:59.308 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:59.017+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:50:59.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:59.305+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:59.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:59.305+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:59.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:50:59.305+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:50:59.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:59.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:59.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:59.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:50:59.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:50:59.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:00.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:59.971+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:00.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:59.971+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:00.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:50:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:50:59.971+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:00.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:00.263+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:00.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:00.263+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:00.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:00.263+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:00.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:00.478+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:00.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:00.478+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:00.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:00.478+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:01.258 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:00.981+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:01.259 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:00.981+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:01.259 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:00.981+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:01.512 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:01.256+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:01.512 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:01.256+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:01.512 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:01.256+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:01.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:01.510+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:01.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:01.510+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:01.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:01.510+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:02.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:01.981+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 361 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:02.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:01.941+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:02.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:01.941+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:02.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:01.941+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:02.536 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:02.237+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:02.536 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:02.237+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:02.536 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:02.237+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:02.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:02.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:02.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:02.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:02.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:02.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:03.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:02.933+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:03.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:02.933+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:03.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:02.933+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:03.527 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:03.252+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:03.527 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:03.252+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:03.527 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:03.252+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:03.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:03.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:03.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:04.282 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:03.982+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:04.282 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:03.982+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:04.282 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:03.982+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:04.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:04.280+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:04.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:04.280+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:04.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:04.280+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:04.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:04.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:04.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:04.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:04.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:04.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:05.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:04.996+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:05.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:04.997+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:05.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:04.997+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:05.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:05.264+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:05.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:05.264+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:05.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:05.264+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:05.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:05.535+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:05.535+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:05.535+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:06.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:05.998+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:06.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:05.998+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:06.278 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:05.998+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:06.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:06.276+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:06.561 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:06.276+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:06.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:06.276+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:06.562 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:06.277+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:06.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:06.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:06.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:06.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:06.561+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:07.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:06.981+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 391 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:07.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:07.021+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:07.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:07.021+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:07.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:07.021+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:07.300 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:07.022+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:07.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:07.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:07.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:07.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:07.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:07.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:07.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:07.538+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:07.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:07.298+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:07.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:07.298+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:07.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:07.298+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:07.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:07.299+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:08.301 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:07.998+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:08.301 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:07.998+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:08.302 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:07.998+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:08.302 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:07.999+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:08.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:08.299+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:08.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:08.299+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:08.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:08.299+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:08.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:08.300+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:08.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:08.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:08.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:08.586+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:09.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:08.981+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:09.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:08.981+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:09.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:08.981+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:09.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:08.981+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:09.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:09.256+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:09.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:09.256+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:09.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:09.256+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:09.557 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:09.257+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:09.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:09.555+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:09.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:09.555+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:09.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:09.555+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:09.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:09.556+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:10.276 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:10.022+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:10.276 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:10.022+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:10.276 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:10.022+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:10.277 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:10.022+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:10.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:10.274+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:10.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:10.274+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:10.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:10.274+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:10.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:10.276+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:10.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:10.581+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:10.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:10.581+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:10.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:10.581+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:10.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:10.581+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:11.319 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:10.996+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:11.319 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:10.996+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:11.319 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:10.996+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:11.319 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:10.997+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:11.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:11.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:11.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:11.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:11.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:11.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:11.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:11.541+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:11.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:11.317+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:11.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:11.317+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:11.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:11.317+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:11.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:11.318+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:12.330 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:11.982+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 419 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:12.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:12.020+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:12.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:12.020+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:12.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:12.020+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:12.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:12.020+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:12.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:12.520+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:12.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:12.520+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:12.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:12.520+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:12.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:12.521+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:12.328+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:12.328+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:12.328+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:12.329+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:13.350 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:13.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:13.350 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:13.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:13.350 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:13.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:13.350 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:13.071+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:13.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:13.348+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:13.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:13.348+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:13.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:13.348+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:13.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:13.349+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:13.531+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:13.531+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:13.531+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:13.531+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:14.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:14.036+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:14.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:14.036+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:14.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:14.036+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:14.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:14.037+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:14.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:14.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:14.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:14.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:14.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:14.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:14.538+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:14.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:14.340+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:14.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:14.340+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:14.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:14.340+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:14.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:14.341+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:15.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:15.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:15.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:15.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:15.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:15.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:15.323 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:15.032+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:15.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:15.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:15.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:15.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:15.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:15.537+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:15.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:15.538+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:15.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:15.321+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:15.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:15.321+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:15.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:15.321+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:15.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:15.322+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:16.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:16.008+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:16.008+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:16.008+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:16.009+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:16.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:16.289+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:16.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:16.289+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:16.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:16.289+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:16.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:16.289+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:16.492+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:16.492+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:16.492+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:16.493+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:17.256 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:16.982+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 450 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:17.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:17.037+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:17.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:17.037+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:17.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:17.037+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:17.256 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:17.037+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:17.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:17.254+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:17.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:17.254+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:17.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:17.254+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:17.526 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:17.255+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:17.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:17.525+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:17.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:17.525+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:17.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:17.525+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:17.525+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:18.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:18.231+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:18.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:18.231+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:18.231+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:18.232+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:18.062+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:18.062+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:18.062+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:18.063+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:18.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:18.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:18.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:18.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:18.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:18.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:18.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:18.487+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:19.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:19.255+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:19.255+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:19.255+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:19.255+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:19.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:19.075+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:19.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:19.075+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:19.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:19.075+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:19.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:19.076+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:19.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:19.466+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:19.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:19.466+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:19.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:19.466+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:19.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:19.467+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:20.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:20.294+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:20.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:20.294+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:20.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:20.294+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:20.440 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:20.295+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:20.441 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:20.087+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:20.441 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:20.087+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:20.441 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:20.087+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:20.441 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:20.087+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:20.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:20.438+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:20.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:20.438+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:20.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:20.438+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:20.439+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:21.295 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:21.038+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:21.295 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:21.038+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:21.295 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:21.038+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:21.295 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:21.039+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:21.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:21.293+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:21.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:21.293+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:21.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:21.293+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:21.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:21.294+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:21.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:21.408+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:21.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:21.408+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:21.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:21.408+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:21.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:21.409+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:22.291 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:21.983+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 478 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:22.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:22.077+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:22.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:22.077+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:22.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:22.077+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:22.291 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:22.077+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:22.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:22.289+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:22.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:22.289+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:22.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:22.289+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:22.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:22.291+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:22.440+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:22.440+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:22.440+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:22.441+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:23.318 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:23.029+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:23.319 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:23.029+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:23.319 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:23.029+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:23.319 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:23.029+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:23.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:23.317+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:23.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:23.317+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:23.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:23.317+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:23.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:23.317+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:23.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:23.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:23.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:23.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:23.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:23.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:23.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:23.443+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:24.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:23.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:24.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:23.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:24.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:23.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:24.330 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:23.988+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:24.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:24.327+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:24.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:24.327+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:24.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:24.327+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:24.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:24.328+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:24.403+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:24.403+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:24.403+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:24.403+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:25.324 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:24.978+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:25.324 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:24.978+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:25.324 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:24.978+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:25.324 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:24.979+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:25.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:25.322+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:25.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:25.322+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:25.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:25.322+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:25.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:25.323+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:25.429+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:25.429+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:25.429+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:25.430+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:26.276 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:25.951+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:26.276 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:25.951+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:26.276 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:25.951+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:26.276 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:25.951+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:26.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:26.442+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:26.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:26.442+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:26.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:26.442+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:26.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:26.443+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:26.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:26.274+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:26.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:26.274+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:26.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:26.274+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:26.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:26.275+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:27.275 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:26.983+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 512 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:27.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:26.950+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:27.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:26.950+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:27.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:26.950+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:27.275 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:26.951+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:27.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:27.273+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:27.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:27.273+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:27.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:27.273+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:27.275+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:27.456+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:27.456+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:27.456+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:27.457+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:28.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:27.932+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:28.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:27.932+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:28.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:27.932+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:28.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:27.933+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:28.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:28.443+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:28.443+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:28.443+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:28.443+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:28.227+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:28.227+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:28.227+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:28.228+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:29.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:28.905+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:29.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:28.905+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:29.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:28.905+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:29.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:28.906+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:29.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:29.455+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:29.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:29.455+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:29.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:29.455+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:29.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:29.456+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:29.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:29.278+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:29.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:29.278+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:29.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:29.278+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:29.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:29.279+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:30.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:29.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:30.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:29.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:30.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:29.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:30.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:29.865+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:30.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:30.259+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:30.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:30.259+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:30.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:30.259+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:30.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:30.260+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:30.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:30.460+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:30.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:30.460+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:30.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:30.460+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:30.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:30.460+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:31.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:30.909+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:31.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:30.909+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:31.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:30.909+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:31.183 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:30.909+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:31.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:31.447+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:31.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:31.447+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:31.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:31.447+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:31.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:31.448+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:31.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:31.288+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:31.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:31.288+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:31.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:31.288+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:31.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:31.290+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:31.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:31.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:31.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:31.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:31.866+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:31.866+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:32.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:31.983+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 540 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:32.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:32.311+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:32.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:32.311+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:32.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:32.311+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:32.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:32.312+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:32.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:32.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:32.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:32.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:32.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:32.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:32.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:32.418+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:33.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:32.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:33.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:32.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:33.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:32.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:33.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:32.833+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:33.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:33.322+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:33.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:33.322+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:33.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:33.322+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:33.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:33.324+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:33.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:33.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:33.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:33.418+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:34.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:33.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:34.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:33.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:34.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:33.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:34.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:33.804+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:34.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:34.291+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:34.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:34.291+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:34.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:34.291+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:34.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:34.292+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:34.445+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:34.445+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:34.445+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:34.446+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:35.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:34.811+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:34.811+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:34.811+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:34.812+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:35.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:35.322+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:35.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:35.322+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:35.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:35.322+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:35.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:35.322+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:35.399+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:35.399+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:35.399+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:35.400+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:36.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:35.820+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:36.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:35.820+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:35.820+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:35.820+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:36.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:36.349+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:36.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:36.349+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:36.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:36.349+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:36.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:36.350+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:36.390+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:36.390+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:36.390+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:36.391+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:37.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:36.984+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 570 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:37.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:36.869+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:37.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:36.869+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:37.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:36.869+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:37.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:36.869+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:37.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:37.412+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:37.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:37.412+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:37.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:37.412+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:37.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:37.414+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:37.312+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:37.312+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:37.312+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:37.313+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:38.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:37.838+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:38.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:37.838+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:38.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:37.838+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:38.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:37.839+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:38.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:38.450+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:38.450+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:38.450+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:38.451+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:38.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:38.337+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:38.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:38.337+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:38.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:38.337+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:38.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:38.338+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:39.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:38.838+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:39.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:38.838+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:39.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:38.838+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:39.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:38.838+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:39.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:39.415+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:39.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:39.415+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:39.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:39.415+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:39.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:39.416+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:39.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:39.388+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:39.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:39.388+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:39.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:39.388+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:39.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:39.388+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:40.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:39.829+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:40.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:39.829+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:40.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:39.829+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:40.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:39.829+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:40.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:40.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:40.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:40.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:40.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:40.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:40.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:40.418+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:40.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:40.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:40.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:40.431+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:41.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:40.818+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:40.818+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:40.818+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:40.819+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:41.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:41.414+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:41.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:41.414+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:41.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:41.414+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:41.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:41.415+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:41.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:41.475+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:41.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:41.475+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:41.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:41.475+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:41.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:41.476+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:42.111 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:41.984+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 598 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:42.112 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:41.854+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:42.112 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:41.854+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:42.112 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:41.854+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:42.112 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:41.855+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:42.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:42.484+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:42.484+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:42.484+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:42.484+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:42.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:42.410+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:42.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:42.410+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:42.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:42.410+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:42.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:42.412+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:43.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:42.837+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:43.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:42.837+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:43.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:42.837+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:43.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:42.838+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:43.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:43.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:43.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:43.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:43.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:43.484+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:43.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:43.420+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:43.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:43.420+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:43.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:43.420+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:43.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:43.420+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:44.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:43.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:44.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:43.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:44.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:43.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:44.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:43.880+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:44.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:44.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:44.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:44.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:44.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:44.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:44.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:44.460+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:44.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:44.494+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:44.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:44.494+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:44.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:44.494+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:44.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:44.494+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:45.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:44.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:45.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:44.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:45.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:44.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:45.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:44.917+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:45.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:45.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:45.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:45.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:45.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:45.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:45.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:45.450+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:45.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:45.532+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:45.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:45.532+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:45.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:45.532+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:45.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:45.533+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:46.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:45.926+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:46.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:45.926+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:46.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:45.926+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:46.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:45.927+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:46.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:46.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:46.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:46.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:46.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:46.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:46.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:46.545+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:46.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:46.432+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:46.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:46.433+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:46.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:46.433+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:46.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:46.434+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:47.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:46.985+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 628 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:47.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:46.887+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:47.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:46.887+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:47.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:46.887+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:47.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:46.887+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:47.866 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:47.451+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:47.866 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:47.451+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:47.866 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:47.451+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:47.866 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:47.453+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:47.866 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:47.586+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:47.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:47.586+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:47.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:47.586+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:47.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:47.587+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:48.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:47.863+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:48.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:47.863+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:48.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:47.863+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:48.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:47.864+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:48.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:48.591+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:48.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:48.591+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:48.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:48.591+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:48.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:48.592+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:48.901 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:48.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:48.901 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:48.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:48.901 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:48.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:48.901 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:48.473+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:49.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:48.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:49.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:48.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:49.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:48.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:49.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:48.899+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:49.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:49.481+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:49.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:49.481+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:49.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:49.481+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:49.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:49.482+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:49.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:49.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:49.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:49.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:49.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:49.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:49.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:49.572+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:50.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:49.939+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:50.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:49.939+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:50.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:49.939+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:50.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:49.940+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:50.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:50.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:50.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:50.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:50.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:50.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:50.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:50.545+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:50.461+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:50.461+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:50.461+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:50.462+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:50.952+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:50.952+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:50.952+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:50.953+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:51.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:51.501+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:51.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:51.501+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:51.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:51.501+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:51.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:51.502+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:51.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:51.515+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:51.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:51.515+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:51.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:51.515+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:51.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:51.516+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:52.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:51.986+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 656 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:52.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:51.917+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:52.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:51.917+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:52.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:51.917+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:52.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:51.917+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:52.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:52.525+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:52.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:52.525+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:52.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:52.525+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:52.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:52.525+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:52.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:52.545+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:52.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:52.545+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:52.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:52.545+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:52.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:52.546+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:53.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:52.945+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:53.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:52.946+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:53.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:52.946+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:53.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:52.947+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:53.582+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:53.582+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:53.582+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:53.583+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:53.566+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:53.566+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:53.566+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:53.567+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:53.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:53.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:53.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:53.989+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:54.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:54.623+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:54.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:54.623+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:54.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:54.623+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:54.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:54.625+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:54.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:54.607+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:54.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:54.607+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:54.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:54.607+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:54.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:54.609+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:55.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:54.940+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:55.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:54.940+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:55.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:54.941+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:55.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:54.941+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:55.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:55.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:55.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:55.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:55.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:55.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:55.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:55.669+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:55.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:55.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:55.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:55.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:55.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:55.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:55.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:55.593+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:56.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:55.901+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:55.901+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:55.901+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:55.902+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:56.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:56.636+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:56.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:56.636+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:56.636+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:56.636+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:56.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:56.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:56.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:56.625+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:56.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:56.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:56.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:56.868+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:57.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:51:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:51:56.986+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 686 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:51:57.867 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:57.647+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:57.867 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:57.647+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:57.867 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:57.647+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:57.867 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:57.648+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:57.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:57.595+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:57.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:57.595+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:57.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:57.595+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:57.868 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:57.596+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:58.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:57.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:58.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:57.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:58.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:57.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:58.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:57.866+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:58.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:58.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:58.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:58.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:58.591+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:58.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:58.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:58.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:58.824+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:58.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:58.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:58.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:58.651+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:59.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:59.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:59.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:51:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:51:59.615+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:51:59.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:59.790+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:59.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:59.790+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:59.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:59.790+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:51:59.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:51:59.791+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:51:59.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:59.675+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:59.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:59.675+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:59.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:59.675+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:51:59.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:51:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:51:59.676+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:00.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:00.787+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:00.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:00.787+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:00.787+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:00.787+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:00.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:00.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:00.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:00.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:00.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:00.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:00.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:00.641+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:00.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:00.655+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:00.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:00.655+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:00.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:00.655+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:00.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:00.656+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:01.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:01.689+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:01.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:01.689+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:01.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:01.689+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:01.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:01.690+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:01.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:01.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:01.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:01.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:01.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:01.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:01.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:01.665+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:01.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:01.782+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:01.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:01.782+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:01.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:01.782+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:01.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:01.783+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:02.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:01.987+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 718 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:02.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:02.646+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:02.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:02.646+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:02.646+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:02.647+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:02.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:02.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:02.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:02.625+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:02.785+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:02.785+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:02.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:02.785+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:02.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:02.786+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:03.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:03.692+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:03.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:03.692+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:03.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:03.692+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:03.814 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:03.693+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:03.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:03.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:03.814 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:03.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:03.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:03.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:03.815 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:03.659+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:04.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:03.812+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:04.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:03.812+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:04.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:03.812+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:04.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:03.813+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:04.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:04.718+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:04.718+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:04.718+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:04.718+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:04.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:04.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:04.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:04.640+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:04.779+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:04.779+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:04.779+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:04.780+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:05.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:05.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:05.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:05.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:05.689+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:05.786+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:05.786+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:05.786+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:05.787+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:05.698+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:05.698+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:05.698+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:05.699+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:06.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:06.720+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:06.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:06.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:06.721+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:06.651+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:06.651+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:06.651+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:06.652+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:06.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:06.797+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:06.797+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:06.797+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:06.798+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:07.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:06.987+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 748 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:07.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:07.758+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:07.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:07.758+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:07.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:07.758+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:07.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:07.759+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:07.679+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:07.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:07.679+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:07.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:07.679+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:07.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:07.680+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:07.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:07.740+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:07.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:07.740+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:07.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:07.740+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:07.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:07.740+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:08.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:08.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:08.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:08.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:08.696+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:08.717+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:08.717+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:08.717+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:08.718+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:08.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:08.775+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:08.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:08.775+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:08.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:08.775+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:08.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:08.775+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:10.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:09.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:10.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:09.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:10.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:09.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:10.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:09.717+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:10.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:09.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:10.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:09.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:10.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:09.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:10.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:09.730+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:10.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:09.716+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:10.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:09.716+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:10.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:09.716+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:10.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:09.717+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:10.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:10.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:10.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:10.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:10.681+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:10.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:10.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:10.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:10.718+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:10.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:10.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:10.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:10.694+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:11.700+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:11.700+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:11.700+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:11.700+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:11.763+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:11.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:11.763+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:11.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:11.763+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:11.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:11.764+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:11.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:11.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:11.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:11.683+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:12.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:11.988+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 788 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:12.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:12.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:12.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:12.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:12.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:12.696+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:12.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:12.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:12.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:12.815+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:12.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:12.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:12.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:12.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:12.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:12.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:12.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:12.688+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:13.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:13.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:13.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:13.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:13.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:13.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:13.811 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:13.728+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:13.812 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:13.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:13.812 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:13.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:13.812 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:13.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:13.812 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:13.669+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:14.190 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:13.809+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:14.190 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:13.809+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:14.190 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:13.809+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:14.190 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:13.810+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:15.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:14.769+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:15.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:14.769+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:15.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:14.769+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:15.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:14.770+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:14.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:14.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:14.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:14.719+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:14.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:14.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:14.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:14.826+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:16.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:15.829+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:15.829+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:15.829+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:15.830+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:16.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:15.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:16.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:15.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:16.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:15.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:16.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:15.772+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:16.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:15.712+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:16.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:15.712+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:16.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:15.712+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:16.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:15.712+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:16.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:16.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:16.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:16.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:16.990 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:16.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:16.991 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:16.743+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:16.991 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:16.810+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:16.991 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:16.810+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:16.991 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:16.810+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:16.991 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:16.811+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:16.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:16.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:16.991 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:16.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:16.992 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:16.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:16.992 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:16.707+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:17.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:16.988+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 818 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:17.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:17.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:17.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:17.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:17.778+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:17.714+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:17.714+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:17.714+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:17.715+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:17.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:17.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:17.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:17.682+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:19.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:18.801+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:18.801+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:18.801+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:18.802+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:18.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:18.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:18.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:18.744+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:19.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:18.709+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:19.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:18.709+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:19.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:18.709+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:19.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:18.709+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:20.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:19.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:20.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:19.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:19.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:19.794+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:20.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:19.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:20.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:19.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:20.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:19.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:20.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:19.708+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:20.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:19.783+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:20.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:19.783+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:20.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:19.783+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:20.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:19.783+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:20.759+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:20.759+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:20.759+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:20.759+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:20.828+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:20.828+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:20.828+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:20.828+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:21.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:20.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:21.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:20.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:21.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:20.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:21.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:20.707+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:21.790+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:21.790+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:21.790+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:21.791+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:21.748+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:21.748+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:21.748+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:21.749+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:21.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:21.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:21.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:21.689+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:22.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:21.989+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 846 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:23.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:22.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:23.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:22.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:23.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:22.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:23.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:22.708+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:22.769+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:22.769+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:22.769+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:22.771+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:22.759+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:22.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:22.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:22.761+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:23.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:23.749+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:23.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:23.749+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:23.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:23.749+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:23.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:23.749+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:24.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:23.796+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:24.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:23.796+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:24.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:23.796+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:24.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:23.797+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:23.810+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:23.810+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:23.810+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:23.811+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:24.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:24.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:24.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:24.741+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:24.842+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:24.842+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:24.842+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:24.844+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:24.815+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:24.815+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:24.816+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:24.817+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:26.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:25.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:26.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:25.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:26.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:25.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:26.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:25.864+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:26.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:25.752+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:26.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:25.752+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:26.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:25.752+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:26.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:25.752+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:26.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:25.837+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:26.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:25.837+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:26.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:25.837+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:26.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:25.837+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:26.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:26.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:26.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:26.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:26.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:26.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:26.841 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:26.716+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:27.114 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:26.989+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 876 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:27.114 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:26.848+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:27.114 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:26.848+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:27.114 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:26.848+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:27.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:26.849+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:27.115 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:26.839+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:27.115 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:26.839+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:27.115 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:26.839+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:27.115 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:26.840+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:28.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:27.866+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:27.866+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:27.866+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:27.866+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:28.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:27.714+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:28.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:27.714+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:28.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:27.714+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:28.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:27.715+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:28.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:27.871+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:28.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:27.871+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:28.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:27.871+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:28.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:27.871+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:29.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:28.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:29.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:28.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:28.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:28.846+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:28.853+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:28.853+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:28.853+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:28.854+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:29.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:28.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:29.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:28.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:29.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:28.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:29.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:28.728+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:30.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:29.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:29.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:29.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:29.883+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:30.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:29.876+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:30.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:29.876+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:30.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:29.876+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:30.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:29.877+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:30.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:29.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:30.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:29.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:30.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:29.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:30.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:29.751+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:31.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:30.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:30.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:30.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:30.911+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:30.833+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:30.833+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:30.833+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:30.834+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:31.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:30.760+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:31.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:30.760+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:31.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:30.760+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:31.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:30.761+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:31.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:31.789+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:31.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:31.789+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:31.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:31.789+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:31.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:31.790+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:31.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:31.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:31.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:31.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:31.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:31.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:31.914 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:31.789+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:32.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:31.990+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 904 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:31.911+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:31.911+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:31.911+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:31.913+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:32.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:32.824+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:32.925 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:32.824+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:32.925 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:32.824+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:32.925 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:32.825+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:32.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:32.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:32.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:32.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:32.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:32.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:32.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:32.748+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:33.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:32.922+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:33.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:32.922+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:33.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:32.922+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:32.923+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:33.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:33.720+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:33.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:33.720+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:33.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:33.720+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:33.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:33.720+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:34.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:33.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:33.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:33.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:33.904+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:34.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:33.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:34.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:33.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:34.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:33.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:34.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:33.861+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:35.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:34.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:35.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:34.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:35.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:34.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:35.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:34.888+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:34.745+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:34.745+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:34.745+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:34.746+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:34.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:34.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:34.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:34.871+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:35.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:35.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:35.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:35.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:35.832+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:35.855+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:35.855+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:35.855+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:35.857+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:35.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:35.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:35.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:35.714+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:36.992 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:36.788+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:36.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:36.788+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:36.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:36.789+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:36.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:36.790+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:36.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:36.853+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:36.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:36.853+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:36.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:36.853+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:36.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:36.854+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:36.994 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:36.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:36.994 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:36.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:36.994 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:36.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:36.994 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:36.734+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:37.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:36.990+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 934 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:37.779+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:37.779+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:37.779+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:37.780+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:37.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:37.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:37.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:37.791+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:38.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:37.856+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:38.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:37.856+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:38.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:37.856+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:38.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:37.856+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:38.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:38.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:38.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:38.733+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:38.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:38.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:38.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:38.881+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:38.756+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:38.756+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:38.756+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:38.757+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:39.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:39.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:39.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:39.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:39.881+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:39.712+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:39.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:39.712+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:39.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:39.712+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:39.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:39.713+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:39.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:39.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:39.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:39.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:39.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:39.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:39.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:39.685+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:40.724+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:40.724+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:40.724+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:40.725+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:40.843+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:40.843+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:40.843+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:40.844+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:40.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:40.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:40.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:40.673+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:41.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:41.752+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:41.752+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:41.752+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:41.752+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:41.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:41.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:41.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:41.812+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:41.667+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:41.667+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:41.667+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:41.667+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:42.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:41.991+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 962 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:42.780+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:42.780+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:42.780+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:42.781+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:42.827+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:42.827+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:42.827+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:42.827+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:42.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:42.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:42.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:42.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:42.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:42.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:42.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:42.685+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:43.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:43.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:43.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:43.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:43.798 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:43.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:43.799 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:43.733+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:44.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:43.836+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:44.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:43.836+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:44.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:43.836+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:44.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:43.837+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:43.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:43.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:43.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:43.797+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:45.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:44.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:44.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:44.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:44.712+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:44.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:44.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:44.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:44.777+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:45.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:44.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:45.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:44.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:45.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:44.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:45.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:44.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:46.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:45.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:46.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:45.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:46.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:45.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:46.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:45.744+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:45.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:45.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:45.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:45.793+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:46.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:45.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:46.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:45.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:46.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:45.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:46.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:45.753+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:46.993 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:46.765+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:46.994 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:46.765+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:46.994 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:46.765+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:46.994 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:46.765+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:46.994 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:46.769+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:46.994 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:46.769+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:46.994 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:46.769+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:46.994 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:46.771+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:46.994 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:46.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:46.995 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:46.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:46.995 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:46.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:46.995 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:46.700+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:47.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:46.991+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 992 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:48.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:47.789+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:48.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:47.789+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:48.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:47.789+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:48.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:47.789+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:48.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:47.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:48.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:47.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:48.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:47.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:48.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:47.727+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:48.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:47.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:48.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:47.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:48.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:47.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:48.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:47.804+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:49.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:48.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:48.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:48.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:48.779+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:48.836+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:48.836+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:48.836+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:48.837+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:48.767+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:48.767+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:48.767+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:48.767+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:50.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:49.855+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:50.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:49.855+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:50.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:49.855+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:50.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:49.855+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:50.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:49.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:50.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:49.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:50.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:49.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:50.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:49.772+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:50.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:49.756+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:50.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:49.756+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:50.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:49.756+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:50.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:49.757+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:51.054 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:50.722+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:51.055 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:50.722+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:51.055 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:50.722+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:51.055 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:50.723+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:51.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:50.799+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:51.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:50.799+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:51.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:50.799+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:51.055 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:50.799+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:51.055 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:50.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:51.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:50.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:51.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:50.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:51.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:50.830+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:51.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:51.836+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:51.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:51.836+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:51.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:51.836+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:51.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:51.836+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:51.847+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:51.847+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:51.847+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:51.848+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:51.680+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:51.680+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:51.680+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:51.681+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:52.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:51.992+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1020 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:52.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:52.848+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:52.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:52.848+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:52.848+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:52.849+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:52.642+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:52.642+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:52.642+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:52.643+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:52.800+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:52.800+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:52.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:52.800+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:52.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:52.801+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:53.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:53.668+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:53.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:53.668+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:53.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:53.668+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:53.796 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:53.669+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:54.127 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:53.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:54.127 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:53.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:54.127 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:53.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:54.127 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:53.899+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:54.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:53.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:54.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:53.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:54.128 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:53.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:54.128 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:53.795+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:54.894 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:54.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:54.894 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:54.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:54.894 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:54.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:54.894 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:54.629+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:54.894 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:54.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:54.894 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:54.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:54.895 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:54.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:54.895 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:54.798+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:55.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:54.892+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:55.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:54.892+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:55.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:54.892+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:55.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:54.892+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:55.845 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:55.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:55.845 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:55.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:55.845 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:55.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:55.845 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:55.590+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:55.843+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:55.843+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:55.843+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:55.843+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:55.906+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:55.906+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:55.906+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:55.907+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:56.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:56.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:56.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:56.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:56.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:56.846+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:56.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:56.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:56.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:56.619+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:57.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:52:56.992+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1050 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:52:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:56.948+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:56.948+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:56.948+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:56.949+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:57.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:57.579+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:57.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:57.579+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:57.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:57.579+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:57.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:57.580+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:58.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:57.815+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:58.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:57.815+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:57.815+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:57.816+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:58.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:57.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:58.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:57.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:58.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:57.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:58.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:57.931+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:58.853 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:58.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:58.853 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:58.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:58.854 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:58.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:58.854 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:58.586+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:52:59.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:58.851+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:59.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:58.851+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:58.851+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:52:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:58.852+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:59.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:58.944+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:59.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:58.944+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:59.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:58.944+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:52:59.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:58.944+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:52:59.870 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:59.549+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:59.870 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:59.549+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:59.870 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:59.549+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:52:59.870 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:52:59.550+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:00.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:59.868+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:00.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:59.868+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:00.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:59.868+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:00.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:52:59.869+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:59.987+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:59.987+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:59.987+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:52:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:52:59.988+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:00.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:00.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:00.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:00.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:00.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:00.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:00.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:00.586+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:01.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:00.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:00.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:00.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:00.919+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:01.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:01.021+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:01.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:01.021+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:01.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:01.021+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:01.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:01.022+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:01.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:01.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:01.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:01.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:01.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:01.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:01.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:01.590+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:02.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:01.993+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1078 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:01.874+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:01.874+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:01.874+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:01.875+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:02.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:01.992+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:02.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:01.992+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:02.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:01.992+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:02.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:01.992+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:02.846+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:02.846+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:02.846+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:02.847+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:02.633+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:02.633+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:02.633+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:02.634+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:03.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:02.967+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:03.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:02.967+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:03.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:02.968+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:03.342 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:02.969+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:03.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:03.615+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:03.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:03.615+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:03.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:03.615+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:03.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:03.617+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:04.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:03.944+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:04.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:03.945+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:04.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:03.945+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:04.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:03.945+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:04.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:03.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:04.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:03.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:04.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:03.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:04.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:03.890+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:04.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:04.602+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:04.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:04.602+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:04.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:04.602+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:04.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:04.603+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:05.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:04.946+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:05.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:04.946+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:05.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:04.946+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:05.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:04.947+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:04.860+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:04.860+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:05.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:04.860+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:05.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:04.861+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:05.898 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:05.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:05.898 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:05.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:05.898 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:05.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:05.898 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:05.560+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:06.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:05.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:05.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:05.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:05.897+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:06.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:05.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:06.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:05.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:06.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:05.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:06.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:05.931+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:06.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:06.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:06.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:06.601+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:07.119 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:06.994+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1109 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:07.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:06.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:07.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:06.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:07.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:06.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:07.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:06.854+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:07.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:06.929+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:07.120 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:06.929+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:07.120 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:06.929+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:07.120 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:06.930+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:07.810+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:07.810+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:07.810+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:07.810+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:07.638+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:07.638+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:07.638+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:07.639+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:07.969+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:07.969+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:07.969+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:07.969+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:08.809+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:08.809+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:08.809+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:08.810+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:08.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:08.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:08.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:08.625+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:09.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:09.017+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:09.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:09.017+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:09.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:09.017+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:09.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:09.018+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:09.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:09.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:09.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:09.775+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:09.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:09.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:09.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:09.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:09.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:09.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:09.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:09.629+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:10.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:10.061+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:10.061+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:10.061+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:10.061+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:10.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:10.595+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:10.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:10.595+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:10.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:10.595+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:10.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:10.597+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:10.793+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:10.793+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:10.793+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:10.794+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:11.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:11.024+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:11.024+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:11.024+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:11.025+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:11.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:11.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:11.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:11.793+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:11.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:11.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:11.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:11.637+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:12.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:12.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:12.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:12.071+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:12.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:11.995+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1154 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:12.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:12.677+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:12.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:12.677+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:12.817 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:12.678+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:12.818 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:12.678+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:13.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:13.026+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:13.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:13.026+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:13.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:13.026+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:13.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:13.026+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:12.815+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:12.815+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:12.815+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:12.816+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:13.798 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:13.657+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:13.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:13.657+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:13.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:13.657+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:13.799 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:13.658+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:14.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:13.984+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:14.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:13.984+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:14.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:13.984+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:14.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:13.985+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:13.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:13.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:13.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:13.797+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:14.690+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:14.690+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:14.690+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:14.691+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:14.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:14.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:14.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:14.755+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:15.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:15.029+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:15.029+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:15.029+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:15.029+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:15.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:15.676+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:15.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:15.676+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:15.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:15.676+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:15.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:15.677+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:15.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:15.770+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:15.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:15.770+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:15.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:15.770+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:15.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:15.771+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:15.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:15.993+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:15.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:15.993+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:15.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:15.993+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:15.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:15.993+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:16.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:16.677+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:16.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:16.677+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:16.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:16.677+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:16.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:16.678+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:16.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:16.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:16.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:16.728+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:17.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:16.995+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1201 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:17.034+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:17.034+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:17.034+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:17.035+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:18.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:17.715+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:18.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:17.715+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:18.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:17.715+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:18.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:17.715+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:18.077 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:17.762+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:18.077 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:17.762+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:18.077 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:17.762+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:18.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:17.763+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:18.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:53:18.122+0000 7f95b0cbb700 0 monclient(hunting): authenticate timed out after 300 2023-12-19T18:53:18.125 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-19T18:53:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:18.074+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:18.074+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:18.074+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:18.075+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:18.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:18.747+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:18.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:18.747+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:18.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:18.747+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:18.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:18.748+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:18.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:18.751+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:18.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:18.751+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:18.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:18.751+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:18.936 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:18.751+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:18.959 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph orch ps' 2023-12-19T18:53:19.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:19.087+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:19.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:19.087+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:19.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:19.087+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:19.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:19.087+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:19.240 INFO:teuthology.orchestra.run.smithi059.stderr:Inferring config /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/mon.smithi059/config 2023-12-19T18:53:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:19.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:19.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:19.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:19.711+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:19.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:19.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:19.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:19.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:19.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:19.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:19.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:19.710+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:20.373 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:20.068+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:20.373 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:20.068+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:20.373 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:20.068+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:20.373 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:20.068+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:20.694+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:20.694+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:20.694+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:20.695+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:20.728+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:20.728+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:20.728+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:20.729+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:21.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:21.043+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:21.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:21.043+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:21.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:21.043+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:21.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:21.044+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:21.664+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:21.664+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:21.664+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:21.665+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:21.762+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:21.762+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:21.762+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:21.762+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:22.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:21.996+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1246 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:22.058+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:22.058+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:22.058+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:22.059+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:23.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:22.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:23.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:22.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:23.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:22.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:23.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:22.769+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:23.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:22.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:23.098 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:22.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:23.098 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:22.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:23.098 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:22.711+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:23.359 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:23.095+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:23.360 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:23.095+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:23.360 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:23.095+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:23.360 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:23.095+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:24.063 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:23.709+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:24.063 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:23.709+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:24.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:23.709+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:24.064 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:23.709+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:24.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:23.758+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:24.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:23.758+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:24.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:23.758+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:24.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:23.759+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:24.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:24.061+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:24.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:24.061+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:24.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:24.061+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:24.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:24.062+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:24.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:24.675+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:24.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:24.675+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:24.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:24.675+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:24.792 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:24.676+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:25.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:24.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:25.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:24.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:25.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:24.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:25.084 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:24.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:25.081+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:25.081+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:25.081+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:25.082+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:25.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:25.824+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:25.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:25.824+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:25.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:25.824+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:25.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:25.825+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:25.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:25.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:25.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:25.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:25.627+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:26.119+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:26.119+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:26.119+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:26.119+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:26.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:26.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:26.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:26.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:26.623+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:26.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:26.805+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:26.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:26.805+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:26.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:26.805+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:26.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:26.806+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:27.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:26.997+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1293 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:27.126+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:27.126+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:27.126+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:27.127+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:27.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:27.599+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:27.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:27.599+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:27.599+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:27.600+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:27.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:27.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:27.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:27.780+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:28.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:28.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:28.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:28.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:28.125+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:28.591+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:28.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:28.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:28.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:28.592+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:28.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:28.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:28.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:28.765+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:29.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:29.106+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:29.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:29.106+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:29.106+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:29.107+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:29.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:29.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:29.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:29.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:29.548+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:29.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:29.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:29.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:29.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:30.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:30.136+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:30.136+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:30.136+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:30.136+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:30.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:30.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:30.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:30.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:30.581+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:30.787+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:30.787+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:30.787+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:30.788+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:31.150+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:31.150+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:31.150+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:31.150+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:31.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:31.605+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:31.605+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:31.605+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:31.607+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:31.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:31.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:31.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:31.774+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:32.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:31.997+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1396 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:32.189+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:32.189+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:32.189+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:32.190+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:32.823 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:32.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:32.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:32.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:32.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:32.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:32.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:32.564+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:33.175 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:32.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:33.175 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:32.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:33.175 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:32.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:33.175 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:32.822+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:33.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:33.173+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:33.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:33.173+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:33.173+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:33.174+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:33.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:33.571+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:33.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:33.571+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:33.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:33.571+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:33.827 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:33.572+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:34.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:33.832+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:34.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:33.832+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:34.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:33.832+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:34.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:33.833+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:34.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:34.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:34.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:34.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:34.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:34.157+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:34.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:34.603+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:34.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:34.603+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:34.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:34.603+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:34.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:34.604+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:35.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:34.847+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:35.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:34.847+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:35.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:34.847+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:35.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:34.848+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:35.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:35.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:35.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:35.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:35.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:35.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:35.583 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:35.204+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:35.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:35.581+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:35.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:35.581+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:35.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:35.581+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:35.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:35.582+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:36.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:35.845+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:36.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:35.845+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:36.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:35.845+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:36.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:35.846+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:36.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:36.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:36.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:36.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:36.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:36.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:36.628 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:36.245+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:36.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:36.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:36.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:36.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:36.627+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:36.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:36.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:36.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:36.850+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:37.254 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:36.998+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1492 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:37.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:37.252+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:37.626 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:37.252+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:37.626 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:37.252+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:37.626 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:37.253+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:37.624+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:37.624+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:37.624+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:37.624+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:37.831+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:37.831+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:37.831+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:37.832+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:38.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:38.288+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:38.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:38.288+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:38.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:38.288+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:38.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:38.289+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:38.654+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:38.654+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:38.654+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:38.655+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:38.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:38.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:38.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:38.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:38.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:38.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:38.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:38.823+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:39.646 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:39.289+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:39.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:39.289+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:39.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:39.289+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:39.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:39.290+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:39.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:39.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:39.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:39.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:39.644+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:39.834+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:39.834+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:39.834+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:39.835+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:40.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:40.262+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:40.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:40.262+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:40.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:40.262+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:40.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:40.262+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:40.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:40.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:40.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:40.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:40.611+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:40.825+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:40.825+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:40.825+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:40.826+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:41.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:41.251+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:41.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:41.251+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:41.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:41.251+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:41.606 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:41.251+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:41.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:41.602+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:41.602+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:41.602+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:41.603+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:41.837+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:41.837+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:41.837+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:41.838+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:42.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:41.999+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1587 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:42.206+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:42.206+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:42.206+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:42.207+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:42.883 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:42.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:42.883 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:42.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:42.884 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:42.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:42.884 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:42.611+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:43.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:42.881+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:43.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:42.881+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:43.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:42.881+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:43.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:42.882+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:43.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:43.239+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:43.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:43.239+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:43.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:43.239+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:43.540 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:43.240+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:43.855 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:43.657+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:43.855 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:43.657+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:43.855 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:43.657+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:43.855 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:43.658+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:44.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:43.853+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:44.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:43.853+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:44.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:43.853+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:44.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:43.854+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:44.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:44.228+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:44.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:44.228+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:44.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:44.228+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:44.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:44.229+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:44.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:44.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:44.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:44.684+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:44.900+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:44.900+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:44.900+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:44.902+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:45.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:45.225+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:45.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:45.225+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:45.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:45.225+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:45.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:45.226+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:45.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:45.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:45.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:45.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:45.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:45.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:45.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:45.644+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:46.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:45.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:45.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:45.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:45.905+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:46.601 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:46.228+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:46.601 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:46.228+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:46.601 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:46.228+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:46.602 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:46.229+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:46.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:46.599+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:46.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:46.599+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:46.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:46.599+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:46.930 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:46.600+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:47.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:46.999+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1683 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:46.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:46.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:46.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:46.929+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:47.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:47.229+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:47.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:47.229+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:47.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:47.229+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:47.555 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:47.230+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:47.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:47.552+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:47.552+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:47.552+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:47.553+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:48.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:47.966+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:48.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:47.966+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:48.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:47.966+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:48.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:47.967+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:48.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:48.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:48.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:48.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:48.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:48.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:48.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:48.241+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:48.931 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:48.546+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:48.931 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:48.546+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:48.932 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:48.546+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:48.932 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:48.546+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:49.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:48.929+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:49.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:48.929+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:49.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:48.929+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:49.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:48.930+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:49.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:49.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:49.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:49.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:49.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:49.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:49.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:49.202+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:49.595+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:49.595+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:49.595+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:49.596+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:50.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:49.948+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:49.948+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:49.948+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:49.948+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:50.610 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:50.200+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:50.611 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:50.200+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:50.611 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:50.200+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:50.611 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:50.201+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:50.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:50.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:50.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:50.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:50.609+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:50.966+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:50.966+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:50.966+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:50.967+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:51.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:51.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:51.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:51.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:51.184+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:51.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:51.658+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:51.658+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:51.658+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:51.659+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:52.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:52.000+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1777 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:51.986+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:51.986+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:51.986+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:52.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:51.986+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:52.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:52.226+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:52.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:52.226+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:52.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:52.226+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:52.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:52.226+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:52.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:52.708+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:52.708+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:52.708+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:52.709+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:53.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:52.950+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:53.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:52.950+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:53.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:52.950+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:53.206 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:52.950+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:53.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:53.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:53.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:53.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:53.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:53.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:53.541 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:53.204+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:53.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:53.687+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:53.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:53.687+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:53.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:53.687+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:53.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:53.688+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:54.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:53.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:54.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:53.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:54.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:53.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:54.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:53.932+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:54.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:54.187+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:54.187+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:54.187+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:54.188+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:54.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:54.680+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:54.680+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:54.680+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:54.680+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:55.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:54.954+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:54.954+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:54.954+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:54.954+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:55.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:55.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:55.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:55.186+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:55.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:55.689+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:55.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:55.690+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:55.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:55.690+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:55.923 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:55.690+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:56.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:55.921+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:56.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:55.921+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:56.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:55.921+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:56.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:55.922+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:56.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:56.214+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:56.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:56.214+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:56.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:56.214+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:56.693 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:56.214+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:56.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:56.879+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:56.879+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:56.879+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:56.880+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:56.690+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:56.690+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:56.690+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:56.692+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:57.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:53:57.001+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1873 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:53:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:57.250+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:57.250+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:57.250+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:57.250+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:57.857+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:57.857+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:57.857+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:57.858+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:57.653+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:57.653+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:57.653+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:57.654+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:58.638 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:58.216+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:58.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:58.216+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:58.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:58.216+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:58.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:58.217+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:58.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:58.858+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:58.858+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:58.858+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:58.858+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:58.648+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:58.648+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:58.648+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:58.649+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:59.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:59.169+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:59.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:59.169+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:59.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:59.169+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:53:59.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:53:59.169+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:53:59.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:59.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:59.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:59.822+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:53:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:53:59.823+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:53:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:59.602+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:59.602+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:59.602+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:53:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:53:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:53:59.602+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:00.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:00.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:00.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:00.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:00.122+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:00.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:00.605+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:00.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:00.605+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:00.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:00.605+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:00.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:00.606+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:01.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:00.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:01.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:00.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:01.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:00.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:01.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:00.864+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:01.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:01.159+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:01.159+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:01.159+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:01.159+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:01.914 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:01.591+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:01.914 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:01.591+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:01.914 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:01.591+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:01.914 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:01.592+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:02.186 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:02.001+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 1967 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:02.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:01.912+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:02.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:01.912+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:02.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:01.912+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:02.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:01.913+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:02.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:02.184+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:02.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:02.184+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:02.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:02.184+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:02.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:02.184+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:02.884 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:02.549+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:02.885 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:02.549+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:02.885 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:02.549+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:02.885 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:02.550+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:03.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:02.882+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:03.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:02.882+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:03.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:02.882+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:03.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:02.883+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:03.157+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:03.157+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:03.157+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:03.158+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:03.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:03.537+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:03.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:03.537+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:03.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:03.537+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:03.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:03.538+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:04.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:03.867+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:04.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:03.867+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:04.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:03.867+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:04.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:03.868+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:04.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:04.176+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:04.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:04.176+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:04.176+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:04.177+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:04.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:04.587+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:04.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:04.587+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:04.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:04.587+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:04.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:04.588+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:05.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:04.905+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:05.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:04.905+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:05.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:04.905+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:05.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:04.906+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:05.175+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:05.175+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:05.175+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:05.175+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:05.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:05.638+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:05.638+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:05.638+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:05.638+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:05.952+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:05.952+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:05.952+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:05.952+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:06.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:06.177+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:06.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:06.177+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:06.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:06.177+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:06.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:06.178+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:06.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:06.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:06.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:06.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:06.928+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:06.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:06.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:06.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:06.680+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:07.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:07.002+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2064 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:07.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:07.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:07.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:07.169+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:08.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:07.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:08.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:07.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:08.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:07.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:08.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:07.910+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:08.178 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:07.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:08.178 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:07.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:08.178 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:07.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:08.178 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:07.724+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:08.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:08.175+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:08.175+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:08.175+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:08.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:08.176+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:08.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:08.689+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:08.689+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:08.689+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:08.690+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:09.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:09.191+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:09.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:09.191+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:09.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:09.191+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:09.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:09.192+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:09.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:08.950+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:09.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:08.950+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:09.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:08.950+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:09.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:08.951+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:10.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:09.702+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:10.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:09.702+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:10.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:09.702+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:10.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:09.704+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:10.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:09.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:10.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:09.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:10.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:09.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:10.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:09.912+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:10.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:10.237+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:10.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:10.237+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:10.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:10.237+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:10.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:10.238+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:11.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:10.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:11.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:10.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:11.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:10.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:11.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:10.951+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:10.732+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:10.732+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:10.732+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:10.734+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:11.642 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:11.275+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:11.643 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:11.275+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:11.643 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:11.275+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:11.643 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:11.276+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:11.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:11.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:11.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:11.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:11.685+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:12.279 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:12.003+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2158 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:12.280 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:11.951+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:12.280 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:11.951+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:12.280 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:11.951+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:12.280 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:11.952+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:12.667 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:12.277+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:12.667 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:12.277+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:12.667 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:12.277+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:12.667 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:12.278+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:12.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:12.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:12.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:12.666+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:13.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:12.996+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:13.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:12.996+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:13.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:12.996+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:13.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:12.998+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:13.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:13.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:13.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:13.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:13.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:13.236+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:13.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:13.653+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:13.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:13.653+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:13.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:13.653+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:13.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:13.654+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:13.998+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:13.998+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:13.998+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:13.999+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:14.241+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:14.241+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:14.241+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:14.241+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:14.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:14.644+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:14.644+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:14.644+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:14.644+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:15.216+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:15.216+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:15.216+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:15.217+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:15.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:14.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:15.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:14.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:15.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:14.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:15.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:14.986+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:15.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:15.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:15.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:15.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:15.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:15.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:15.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:15.683+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:16.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:16.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:16.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:16.186+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:16.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:16.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:16.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:16.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:16.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:16.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:16.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:16.025+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:16.635+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:16.635+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:16.635+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:16.635+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:17.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:17.004+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2255 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:17.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:17.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:17.203+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:17.204+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:17.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:17.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:17.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:17.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:17.057+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:17.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:17.611+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:17.611+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:17.611+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:17.612+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:18.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:18.010+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:18.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:18.010+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:18.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:18.010+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:18.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:18.011+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:18.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:18.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:18.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:18.184+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:18.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:18.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:18.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:18.595+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:19.056+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:19.056+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:19.056+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:19.057+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:19.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:19.197+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:19.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:19.197+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:19.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:19.197+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:19.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:19.198+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:19.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:19.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:19.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:19.594+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:20.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:20.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:20.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:20.155+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:20.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:20.086+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:20.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:20.086+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:20.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:20.086+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:20.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:20.087+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:20.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:20.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:20.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:20.641+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:21.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:21.058+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:21.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:21.058+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:21.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:21.058+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:21.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:21.059+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:21.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:21.143+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:21.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:21.143+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:21.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:21.143+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:21.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:21.143+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:21.655+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:21.655+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:21.655+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:21.656+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:22.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:22.004+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2348 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:22.146+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:22.146+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:22.146+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:22.147+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:22.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:22.095+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:22.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:22.095+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:22.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:22.095+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:22.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:22.097+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:22.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:22.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:22.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:22.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:22.681+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:23.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:23.133+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:23.133+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:23.133+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:23.134+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:23.125+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:23.125+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:23.125+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:23.127+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:24.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:23.708+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:24.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:23.708+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:24.097 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:23.708+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:24.098 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:23.709+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:24.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:24.095+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:24.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:24.095+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:24.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:24.095+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:24.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:24.096+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:24.135+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:24.135+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:24.135+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:24.136+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:24.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:24.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:24.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:24.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:24.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:24.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:24.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:24.661+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:25.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:25.168+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:25.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:25.168+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:25.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:25.168+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:25.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:25.169+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:25.066+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:25.066+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:25.066+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:25.066+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:25.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:25.663+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:25.663+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:25.663+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:25.664+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:26.069+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:26.069+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:26.069+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:26.069+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:26.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:26.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:26.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:26.120+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:26.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:26.628+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:26.628+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:26.628+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:26.628+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:27.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:27.005+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2445 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:27.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:27.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:27.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:27.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:27.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:27.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:27.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:27.137+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:27.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:27.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:27.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:27.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:27.033+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:27.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:27.656+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:27.656+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:27.656+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:27.657+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:28.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:28.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:28.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:28.119+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:28.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:27.997+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:28.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:27.997+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:28.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:27.997+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:28.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:27.998+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:28.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:28.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:28.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:28.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:28.684+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:29.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:29.158+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:29.158+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:29.158+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:29.158+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:29.033+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:29.033+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:29.033+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:29.034+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:29.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:29.667+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:29.667+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:29.667+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:29.667+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:30.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:30.184+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:30.184+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:30.184+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:30.185+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:30.071+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:30.071+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:30.071+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:30.072+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:30.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:30.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:30.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:30.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:30.682+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:31.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:31.220+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:31.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:31.220+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:31.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:31.220+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:31.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:31.220+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:31.100+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:31.100+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:31.100+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:31.101+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:31.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:31.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:31.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:31.716+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:32.430 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:32.006+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2538 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:32.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:32.189+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:32.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:32.189+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:32.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:32.189+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:32.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:32.191+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:32.430 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:32.123+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:32.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:32.123+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:32.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:32.123+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:32.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:32.124+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:32.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:32.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:32.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:32.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:32.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:32.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:32.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:32.669+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:33.135+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:33.135+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:33.135+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:33.135+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:33.181+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:33.181+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:33.181+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:33.182+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:33.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:33.637+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:33.637+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:33.637+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:33.638+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:34.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:34.203+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:34.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:34.203+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:34.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:34.203+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:34.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:34.205+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:34.101+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:34.101+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:34.101+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:34.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:34.101+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:34.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:34.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:34.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:34.668+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:35.062+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:35.062+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:35.062+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:35.062+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:35.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:35.191+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:35.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:35.191+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:35.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:35.191+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:35.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:35.192+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:35.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:35.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:35.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:35.696+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:36.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:36.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:36.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:36.166+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:36.037+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:36.037+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:36.037+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:36.038+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:37.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:36.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:37.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:36.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:37.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:36.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:37.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:36.718+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:37.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:37.006+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2635 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:37.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:37.146+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:37.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:37.146+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:37.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:37.146+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:37.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:37.146+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:37.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:37.015+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:37.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:37.015+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:37.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:37.015+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:37.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:37.016+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:38.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:37.700+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:38.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:37.700+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:38.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:37.700+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:38.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:37.702+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:38.015+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:38.015+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:38.015+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:38.016+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:38.170+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:38.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:38.170+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:38.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:38.170+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:38.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:38.170+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:38.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:38.655+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:38.655+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:38.655+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:38.656+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:39.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:39.208+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:39.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:39.208+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:39.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:39.208+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:39.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:39.209+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:39.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:39.025+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:39.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:39.025+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:39.454 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:39.025+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:39.455 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:39.026+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:40.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:39.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:40.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:39.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:40.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:39.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:40.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:39.700+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:40.183+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:40.183+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:40.183+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:40.184+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:40.054+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:40.054+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:40.054+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:40.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:40.054+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:41.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:40.719+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:41.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:40.719+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:41.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:40.719+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:41.056 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:40.721+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:41.054+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:41.054+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:41.054+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:41.054+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:41.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:41.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:41.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:41.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:41.138+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:41.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:41.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:41.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:41.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:41.717+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:42.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:42.007+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2728 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:42.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:42.100+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:42.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:42.100+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:42.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:42.100+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:42.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:42.100+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:42.025+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:42.025+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:42.025+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:42.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:42.025+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:42.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:42.677+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:42.677+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:42.677+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:42.678+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:43.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:43.099+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:43.099+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:43.099+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:43.100+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:43.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:43.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:43.070+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:43.071+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:43.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:43.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:43.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:43.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:43.728+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:44.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:44.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:44.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:44.121+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:44.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:44.079+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:44.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:44.079+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:44.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:44.079+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:44.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:44.080+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:45.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:44.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:45.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:44.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:45.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:44.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:45.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:44.718+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:45.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:45.068+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:45.068+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:45.068+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:45.070+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:45.111+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:45.111+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:45.111+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:45.112+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:45.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:45.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:45.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:45.687+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:45.688+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:46.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:46.047+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:46.047+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:46.047+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:46.048+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:46.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:46.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:46.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:46.156+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:46.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:46.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:46.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:46.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:46.662+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:47.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:47.007+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2825 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:47.141+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:47.141+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:47.141+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:47.142+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:47.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:47.086+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:47.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:47.086+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:47.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:47.086+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:47.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:47.087+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:47.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:47.645+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:47.645+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:47.645+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:47.646+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:48.092+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:48.092+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:48.092+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:48.093+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:48.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:48.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:48.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:48.068+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:48.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:48.646+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:48.646+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:48.646+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:48.647+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:49.140+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:49.141+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:49.141+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:49.141+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:49.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:49.046+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:49.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:49.046+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:49.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:49.046+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:49.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:49.047+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:49.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:49.615+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:49.615+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:49.615+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:49.616+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:50.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:50.136+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:50.136+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:50.136+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:50.136+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:50.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:50.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:50.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:50.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:50.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:50.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:50.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:50.024+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:50.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:50.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:50.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:50.622+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:51.174+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:51.174+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:51.174+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:51.175+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:51.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:51.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:51.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:51.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:51.028+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:51.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:51.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:51.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:51.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:51.590+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:52.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:52.008+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 2918 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:52.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:52.206+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:52.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:52.206+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:52.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:52.206+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:52.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:52.206+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:52.009+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:52.009+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:52.009+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:52.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:52.010+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:52.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:52.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:52.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:52.590+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:52.992+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:52.992+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:52.992+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:52.993+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:53.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:53.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:53.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:53.235+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:53.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:53.636+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:53.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:53.636+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:53.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:53.636+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:53.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:53.636+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:54.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:53.999+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:54.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:53.999+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:54.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:53.999+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:54.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:53.999+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:54.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:54.272+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:54.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:54.272+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:54.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:54.272+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:54.664 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:54.272+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:54.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:54.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:54.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:54.662+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:54.990+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:54.990+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:54.990+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:54.991+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:55.225+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:55.225+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:55.225+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:55.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:55.226+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:55.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:55.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:55.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:55.684+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:56.273 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:55.964+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:56.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:55.964+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:56.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:55.964+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:56.274 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:55.965+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:56.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:56.271+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:56.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:56.271+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:56.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:56.271+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:56.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:56.272+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:57.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:56.959+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:57.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:56.960+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:57.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:56.960+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:57.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:56.960+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:57.012 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:56.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:57.012 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:56.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:57.012 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:56.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:57.012 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:56.722+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:57.267 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:54:57.009+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3015 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:54:57.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:57.265+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:57.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:57.265+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:57.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:57.265+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:57.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:57.266+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:57.961+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:57.961+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:57.961+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:57.963+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:57.735+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:57.735+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:57.735+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:57.736+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:58.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:58.292+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:58.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:58.292+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:58.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:58.292+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:58.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:58.293+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:58.937+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:58.937+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:58.937+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:54:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:58.937+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:54:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:58.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:58.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:59.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:58.700+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:54:59.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:58.700+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:54:59.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:59.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:59.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:59.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:59.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:59.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:54:59.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:54:59.249+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:00.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:59.962+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:00.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:59.962+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:00.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:59.962+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:00.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:54:59.963+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:00.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:59.731+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:00.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:59.731+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:00.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:59.731+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:00.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:54:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:54:59.731+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:00.269+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:00.269+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:00.269+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:00.270+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:01.008 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:00.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:01.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:00.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:01.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:00.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:01.009 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:00.717+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:01.277 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:01.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:01.278 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:01.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:01.278 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:01.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:01.278 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:01.007+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:01.646 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:01.274+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:01.646 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:01.274+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:01.646 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:01.274+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:01.646 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:01.275+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:01.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:01.755+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:01.755+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:01.755+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:01.756+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:02.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:02.009+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3108 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:02.294 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:02.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:02.294 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:02.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:02.294 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:02.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:02.294 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:02.024+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:02.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:02.291+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:02.291+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:02.291+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:02.292+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:03.073 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:02.764+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:03.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:02.764+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:03.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:02.764+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:03.074 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:02.764+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:03.311+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:03.311+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:03.311+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:03.312+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:03.071+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:03.071+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:03.071+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:03.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:03.072+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:04.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:03.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:04.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:03.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:04.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:03.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:04.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:03.804+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:04.117+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:04.117+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:04.117+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:04.119+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:04.331+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:04.331+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:04.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:04.331+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:04.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:04.332+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:05.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:04.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:05.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:04.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:05.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:04.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:05.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:04.777+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:05.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:05.317+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:05.317+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:05.317+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:05.317+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:05.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:05.115+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:05.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:05.115+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:05.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:05.115+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:05.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:05.115+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:06.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:05.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:06.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:05.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:06.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:05.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:06.087 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:05.760+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:06.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:06.085+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:06.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:06.085+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:06.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:06.085+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:06.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:06.085+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:06.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:06.356+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:06.356+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:06.356+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:06.357+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:07.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:06.755+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:07.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:06.755+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:07.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:06.755+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:07.013 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:06.756+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:07.375 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:07.010+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3205 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:07.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:07.094+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:07.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:07.094+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:07.375 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:07.094+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:07.376 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:07.095+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:07.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:07.373+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:07.373+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:07.373+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:07.374+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:08.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:07.788+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:08.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:07.788+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:08.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:07.788+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:08.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:07.789+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:08.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:08.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:08.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:08.137+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:08.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:08.383+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:08.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:08.383+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:08.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:08.383+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:08.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:08.384+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:09.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:08.817+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:09.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:08.817+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:09.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:08.817+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:09.121 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:08.818+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:09.374 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:09.118+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:09.374 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:09.118+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:09.374 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:09.118+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:09.374 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:09.119+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:09.372+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:09.372+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:09.372+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:09.372+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:10.142 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:09.792+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:10.142 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:09.792+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:10.142 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:09.792+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:10.142 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:09.793+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:10.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:10.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:10.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:10.388+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:10.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:10.140+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:10.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:10.140+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:10.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:10.140+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:10.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:10.141+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:11.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:10.749+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:11.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:10.749+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:11.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:10.749+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:11.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:10.750+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:11.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:11.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:11.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:11.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:11.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:11.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:11.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:11.154+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:11.376+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:11.376+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:11.376+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:11.377+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:11.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:11.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:11.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:11.795+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:12.366 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:12.011+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3298 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:12.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:12.183+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:12.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:12.183+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:12.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:12.183+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:12.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:12.184+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:12.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:12.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:12.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:12.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:12.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:12.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:12.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:12.418+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:12.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:12.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:12.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:12.785+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:13.546 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:13.219+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:13.546 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:13.219+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:13.546 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:13.219+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:13.546 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:13.221+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:13.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:13.421+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:13.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:13.421+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:13.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:13.421+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:13.547 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:13.421+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:13.827+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:13.827+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:13.827+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:13.829+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:14.198+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:14.198+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:14.198+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:14.199+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:14.440+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:14.440+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:14.440+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:14.441+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:15.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:14.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:15.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:14.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:15.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:14.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:15.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:14.825+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:15.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:15.192+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:15.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:15.192+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:15.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:15.192+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:15.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:15.193+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:15.866 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:15.474+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:15.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:15.474+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:15.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:15.474+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:15.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:15.475+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:15.864+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:15.864+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:15.864+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:15.865+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:16.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:16.204+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:16.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:16.204+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:16.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:16.204+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:16.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:16.204+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:16.852 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:16.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:16.852 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:16.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:16.853 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:16.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:16.853 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:16.495+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:17.136 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:16.850+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:17.137 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:16.850+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:17.137 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:16.850+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:17.137 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:16.851+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:17.137 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:17.012+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3396 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:17.156+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:17.156+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:17.156+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:17.157+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:17.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:17.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:17.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:17.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:17.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:17.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:17.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:17.505+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:18.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:17.874+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:18.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:17.874+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:18.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:17.874+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:18.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:17.875+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:18.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:18.195+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:18.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:18.195+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:18.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:18.195+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:18.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:18.196+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:18.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:18.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:18.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:18.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:18.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:18.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:18.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:18.506+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:19.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:18.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:19.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:18.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:19.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:18.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:19.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:18.898+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:19.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:19.172+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:19.172+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:19.172+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:19.174+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:19.888 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:19.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:19.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:19.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:19.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:19.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:19.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:19.503+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:20.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:19.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:19.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:19.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:19.887+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:20.491 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:20.217+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:20.491 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:20.217+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:20.491 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:20.217+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:20.491 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:20.218+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:20.909 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:20.489+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:20.909 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:20.489+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:20.909 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:20.489+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:20.909 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:20.489+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:21.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:20.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:20.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:20.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:20.908+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:21.499 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:21.212+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:21.500 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:21.212+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:21.500 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:21.212+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:21.500 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:21.213+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:21.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:21.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:21.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:21.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:21.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:21.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:21.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:21.498+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:22.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:22.013+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3490 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:22.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:21.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:22.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:21.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:22.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:21.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:22.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:21.908+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:22.513 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:22.214+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:22.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:22.214+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:22.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:22.214+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:22.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:22.214+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:22.920 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:22.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:22.920 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:22.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:22.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:22.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:22.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:22.512+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:23.187 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:22.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:23.188 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:22.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:23.188 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:22.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:23.188 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:22.919+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:23.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:23.185+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:23.185+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:23.185+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:23.186+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:23.499+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:23.499+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:23.499+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:23.500+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:23.959+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:23.959+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:23.959+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:23.960+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:24.182+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:24.182+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:24.182+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:24.184+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:24.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:24.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:24.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:24.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:24.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:24.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:24.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:24.488+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:24.938+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:24.938+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:24.938+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:24.940+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:25.502 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:25.232+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:25.502 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:25.232+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:25.503 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:25.232+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:25.503 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:25.232+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:25.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:25.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:25.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:25.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:25.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:25.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:25.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:25.501+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:26.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:25.933+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:26.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:25.933+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:26.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:25.933+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:26.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:25.934+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:26.189+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:26.189+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:26.189+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:26.190+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:26.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:26.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:26.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:26.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:26.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:26.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:26.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:26.488+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:27.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:27.014+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3587 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:27.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:26.941+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:27.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:26.941+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:27.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:26.941+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:27.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:26.942+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:27.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:27.152+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:27.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:27.152+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:27.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:27.152+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:27.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:27.153+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:27.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:27.531+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:27.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:27.531+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:27.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:27.531+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:27.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:27.531+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:28.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:27.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:28.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:27.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:28.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:27.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:28.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:27.926+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:28.179+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:28.179+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:28.179+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:28.180+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:28.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:28.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:28.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:28.503+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:29.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:28.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:29.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:28.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:29.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:28.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:29.247 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:28.951+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:29.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:29.180+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:29.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:29.180+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:29.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:29.180+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:29.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:29.182+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:29.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:29.463+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:29.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:29.463+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:29.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:29.463+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:29.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:29.463+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:30.162 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:29.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:30.162 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:29.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:30.162 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:29.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:30.163 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:29.903+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:30.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:30.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:30.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:30.161+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:30.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:30.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:30.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:30.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:30.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:30.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:30.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:30.485+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:31.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:31.187+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:31.187+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:31.187+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:31.188+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:30.943+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:30.943+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:30.943+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:30.944+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:31.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:31.466+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:31.466+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:31.466+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:31.467+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:32.426 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:32.015+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3679 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:32.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:32.197+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:32.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:32.197+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:32.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:32.197+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:32.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:32.198+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:32.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:31.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:32.426 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:31.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:32.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:31.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:32.427 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:31.957+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:32.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:32.423+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:32.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:32.423+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:32.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:32.423+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:32.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:32.424+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:33.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:32.929+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:32.929+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:32.929+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:32.930+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:33.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:33.214+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:33.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:33.214+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:33.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:33.214+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:33.548 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:33.215+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:33.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:33.396+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:33.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:33.396+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:33.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:33.396+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:33.548 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:33.397+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:33.895+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:33.895+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:33.895+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:33.896+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:34.198+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:34.198+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:34.198+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:34.199+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:34.356+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:34.356+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:34.356+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:34.356+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:35.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:34.905+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:35.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:34.905+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:35.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:34.905+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:35.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:34.906+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:35.191+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:35.191+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:35.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:35.191+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:35.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:35.192+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:35.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:35.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:35.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:35.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:35.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:35.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:35.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:35.392+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:36.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:35.877+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:36.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:35.877+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:36.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:35.877+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:36.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:35.878+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:36.170+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:36.170+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:36.170+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:36.171+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:36.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:36.396+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:36.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:36.396+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:36.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:36.396+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:36.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:36.397+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:37.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:37.015+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3776 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:36.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:36.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:36.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:36.903+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:37.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:37.134+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:37.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:37.134+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:37.134+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:37.135+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:37.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:37.351+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:37.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:37.351+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:37.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:37.351+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:37.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:37.351+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:38.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:37.891+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:38.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:37.891+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:38.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:37.891+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:38.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:37.892+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:38.375+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:38.375+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:38.375+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:38.376+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:38.151+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:38.151+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:38.151+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:38.153+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:39.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:38.876+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:39.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:38.876+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:39.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:38.876+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:39.184 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:38.877+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:39.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:39.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:39.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:39.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:39.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:39.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:39.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:39.392+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:39.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:39.182+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:39.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:39.182+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:39.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:39.182+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:39.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:39.183+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:40.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:39.835+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:40.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:39.835+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:40.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:39.835+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:40.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:39.836+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:40.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:40.418+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:40.418+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:40.418+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:40.418+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:40.177+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:40.177+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:40.177+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:40.178+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:41.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:40.813+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:41.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:40.813+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:41.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:40.813+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:41.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:40.814+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:41.191+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:41.191+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:41.191+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:41.192+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:41.436+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:41.436+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:41.436+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:41.436+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:41.811+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:41.811+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:41.811+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:41.812+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:42.426 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:42.017+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3869 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:42.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:42.236+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:42.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:42.236+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:42.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:42.236+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:42.427 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:42.237+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:42.424+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:42.424+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:42.424+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:42.425+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:43.086 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:42.770+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:43.086 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:42.770+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:43.086 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:42.770+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:43.086 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:42.771+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:43.426 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:43.205+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:43.427 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:43.205+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:43.427 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:43.205+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:43.427 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:43.206+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:43.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:43.424+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:43.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:43.424+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:43.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:43.424+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:43.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:43.425+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:43.792+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:43.792+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:43.792+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:43.793+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:44.423+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:44.423+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:44.423+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:44.423+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:44.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:44.222+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:44.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:44.222+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:44.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:44.222+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:44.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:44.223+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:44.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:44.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:44.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:44.803+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:45.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:45.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:45.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:45.387+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:45.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:45.200+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:45.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:45.200+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:45.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:45.200+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:45.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:45.201+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:46.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:45.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:46.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:45.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:46.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:45.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:46.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:45.826+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:46.348+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:46.348+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:46.348+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:46.349+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:46.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:46.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:46.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:46.166+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:47.144 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:46.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:47.144 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:46.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:47.144 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:46.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:47.145 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:46.815+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:47.145 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:47.017+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 3966 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:47.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:47.159+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:47.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:47.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:47.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:47.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:47.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:47.160+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:47.314+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:47.314+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:47.314+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:47.314+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:48.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:47.836+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:48.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:47.836+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:48.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:47.836+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:48.113 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:47.837+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:48.111+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:48.111+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:48.111+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:48.112+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:48.350+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:48.350+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:48.350+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:48.351+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:49.130 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:48.798+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:49.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:48.798+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:49.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:48.798+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:49.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:48.799+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:49.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:49.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:49.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:49.356+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:49.128+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:49.128+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:49.128+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:49.129+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:50.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:49.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:50.160 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:49.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:50.160 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:49.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:50.160 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:49.803+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:50.388+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:50.388+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:50.388+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:50.389+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:50.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:50.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:50.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:50.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:50.158+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:51.118 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:50.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:51.118 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:50.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:51.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:50.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:51.119 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:50.755+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:51.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:51.116+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:51.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:51.116+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:51.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:51.116+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:51.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:51.117+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:51.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:51.416+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:51.416+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:51.416+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:51.417+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:52.020 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:51.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:52.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:51.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:52.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:51.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:52.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:51.707+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:52.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:52.018+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4059 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:52.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:52.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:52.302 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:52.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:52.303 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:52.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:52.303 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:52.088+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:52.391+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:52.391+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:52.391+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:52.391+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:53.121 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:52.736+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:53.121 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:52.736+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:53.121 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:52.736+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:53.121 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:52.737+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:53.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:53.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:53.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:53.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:53.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:53.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:53.405 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:53.119+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:53.403+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:53.403+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:53.403+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:53.403+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:54.101 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:53.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:54.101 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:53.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:54.101 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:53.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:54.101 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:53.722+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:54.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:54.098+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:54.098+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:54.099+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:54.100+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:54.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:54.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:54.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:54.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:54.451+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:55.137 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:54.722+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:55.138 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:54.722+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:55.138 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:54.722+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:55.138 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:54.724+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:55.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:55.135+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:55.135+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:55.135+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:55.137+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:55.732 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:55.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:55.733 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:55.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:55.733 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:55.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:55.733 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:55.478+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:55.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:55.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:55.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:56.089 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:55.732+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:56.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:56.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:56.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:56.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:56.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:56.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:56.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:56.088+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:56.761 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:56.492+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:56.761 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:56.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:56.761 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:56.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:56.761 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:56.494+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:57.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:56.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:57.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:56.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:57.021 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:56.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:57.022 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:56.760+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:57.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:55:57.019+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4156 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:55:57.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:57.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:57.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:57.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:57.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:57.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:57.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:57.137+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:57.766+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:57.766+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:57.766+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:57.767+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:57.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:57.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:57.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:57.534+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:58.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:58.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:58.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:58.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:58.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:58.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:58.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:58.153+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:58.488+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:58.488+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:58.488+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:58.489+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:58.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:58.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:58.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:58.728+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:59.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:59.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:59.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:55:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:55:59.161+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:55:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:59.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:59.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:59.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:55:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:55:59.494+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:55:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:59.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:59.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:59.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:55:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:55:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:55:59.688+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:00.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:00.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:00.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:00.158+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:00.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:00.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:00.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:00.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:00.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:00.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:00.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:00.680+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:00.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:00.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:00.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:00.452+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:01.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:01.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:01.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:01.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:01.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:01.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:01.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:01.138+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:01.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:01.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:01.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:01.478+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:01.674+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:01.674+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:01.674+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:01.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:01.676+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:02.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:02.020+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4249 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:02.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:02.133+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:02.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:02.133+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:02.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:02.133+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:02.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:02.134+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:02.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:02.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:02.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:02.500+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:02.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:02.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:02.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:02.708+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:03.086+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:03.086+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:03.086+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:03.087+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:03.537+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:03.537+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:03.537+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:03.537+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:03.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:03.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:03.681+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:03.682+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:04.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:04.109+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:04.109+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:04.109+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:04.110+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:04.580+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:04.580+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:04.580+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:04.581+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:04.698+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:04.698+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:04.698+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:04.698+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:05.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:05.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:05.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:05.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:05.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:05.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:05.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:05.106+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:05.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:05.569+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:05.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:05.569+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:05.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:05.569+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:05.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:05.570+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:05.954 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:05.705+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:05.954 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:05.705+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:05.954 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:05.705+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:05.954 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:05.706+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:06.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:06.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:06.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:06.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:06.107+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:06.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:06.527+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:06.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:06.527+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:06.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:06.527+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:06.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:06.528+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:06.685+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:06.685+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:06.685+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:06.686+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:07.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:07.021+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4346 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:07.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:07.142+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:07.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:07.142+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:07.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:07.142+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:07.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:07.144+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:07.531+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:07.532+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:07.532+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:07.725 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:07.532+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:08.106 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:07.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:08.106 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:07.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:08.106 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:07.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:08.106 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:07.724+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:08.104+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:08.104+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:08.104+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:08.104+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:08.749+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:08.749+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:08.749+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:08.751+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:08.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:08.564+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:08.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:08.564+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:08.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:08.564+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:08.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:08.565+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:09.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:09.154+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:09.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:09.154+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:09.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:09.154+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:09.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:09.156+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:09.781+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:09.781+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:09.781+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:09.782+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:09.576+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:09.576+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:09.576+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:09.576+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:10.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:10.193+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:10.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:10.193+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:10.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:10.193+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:10.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:10.194+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:10.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:10.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:10.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:10.804+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:10.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:10.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:10.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:10.587+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:11.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:11.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:11.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:11.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:11.203+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:11.847 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:11.587+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:11.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:11.587+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:11.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:11.587+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:11.848 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:11.587+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:12.147 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:11.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:12.147 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:11.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:12.147 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:11.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:12.147 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:11.846+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:12.148 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:12.022+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4442 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:12.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:12.197+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:12.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:12.197+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:12.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:12.197+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:12.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:12.197+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:12.863 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:12.600+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:12.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:12.600+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:12.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:12.600+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:12.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:12.600+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:13.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:12.861+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:12.861+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:12.861+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:12.862+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:13.551 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:13.238+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:13.551 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:13.238+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:13.552 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:13.238+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:13.552 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:13.239+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:13.806 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:13.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:13.806 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:13.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:13.806 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:13.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:13.806 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:13.647+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:13.843+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:13.843+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:13.843+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:13.844+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:14.257+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:14.257+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:14.257+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:14.258+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:14.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:14.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:14.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:14.879+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:14.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:14.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:14.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:14.697+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:15.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:15.246+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:15.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:15.246+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:15.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:15.246+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:15.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:15.247+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:15.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:15.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:15.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:15.650+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:15.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:15.860+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:15.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:15.860+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:15.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:15.860+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:15.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:15.860+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:16.610 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:16.270+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:16.610 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:16.270+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:16.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:16.270+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:16.611 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:16.271+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:16.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:16.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:16.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:16.831+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:16.608+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:16.608+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:16.608+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:16.609+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:17.313 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:17.023+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4543 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:17.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:17.311+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:17.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:17.311+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:17.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:17.311+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:17.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:17.312+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:17.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:17.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:17.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:17.650+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:17.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:17.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:17.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:17.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:17.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:17.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:17.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:17.815+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:18.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:18.324+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:18.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:18.324+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:18.617 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:18.324+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:18.617 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:18.324+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:18.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:18.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:18.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:18.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:18.779+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:18.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:18.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:18.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:18.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:18.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:18.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:18.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:18.615+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:19.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:19.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:19.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:19.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:19.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:19.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:19.578 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:19.345+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:19.575+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:19.575+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:19.575+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:19.576+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:19.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:19.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:19.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:19.805+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:20.350+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:20.350+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:20.350+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:20.351+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:20.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:20.559+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:20.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:20.559+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:20.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:20.559+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:20.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:20.560+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:20.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:20.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:20.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:20.784+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:21.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:21.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:21.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:21.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:21.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:21.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:21.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:21.380+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:21.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:21.522+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:21.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:21.522+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:21.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:21.522+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:21.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:21.522+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:21.815+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:21.815+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:21.815+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:21.816+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:22.399 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:22.023+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4643 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:22.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:22.397+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:22.397+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:22.397+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:22.397+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:22.532+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:22.532+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:22.532+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:22.533+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:22.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:22.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:22.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:22.797+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:23.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:23.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:23.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:23.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:23.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:23.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:23.550 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:23.443+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:23.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:23.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:23.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:23.546+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:23.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:23.758+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:23.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:23.758+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:23.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:23.758+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:23.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:23.759+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:24.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:24.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:24.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:24.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:24.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:24.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:24.764 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:24.550+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:24.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:24.490+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:24.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:24.490+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:24.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:24.490+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:24.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:24.490+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:24.760+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:24.760+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:24.760+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:24.761+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:25.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:25.499+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:25.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:25.499+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:25.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:25.499+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:25.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:25.499+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:25.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:25.552+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:25.552+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:25.552+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:25.552+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:25.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:25.728+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:25.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:25.728+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:25.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:25.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:25.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:25.730+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:26.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:26.502+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:26.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:26.502+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:26.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:26.502+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:26.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:26.503+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:26.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:26.531+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:26.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:26.531+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:26.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:26.531+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:26.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:26.531+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:27.038 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:27.024+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4744 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:27.038 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:26.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:27.038 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:26.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:27.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:26.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:27.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:26.777+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:27.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:27.459+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:27.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:27.459+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:27.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:27.459+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:27.778 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:27.460+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:27.778 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:27.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:27.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:27.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:27.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:27.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:27.779 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:27.486+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:28.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:27.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:27.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:27.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:27.776+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:28.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:28.456+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:28.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:28.456+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:28.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:28.456+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:28.757 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:28.457+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:28.757 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:28.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:28.757 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:28.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:28.757 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:28.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:28.758 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:28.452+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:29.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:28.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:28.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:28.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:28.755+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:29.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:29.429+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:29.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:29.429+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:29.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:29.429+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:29.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:29.430+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:29.438+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:29.438+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:29.438+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:29.438+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:29.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:29.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:29.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:29.760+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:30.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:30.408+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:30.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:30.408+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:30.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:30.408+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:30.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:30.409+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:30.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:30.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:30.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:30.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:30.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:30.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:30.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:30.394+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:30.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:30.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:30.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:30.803+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:31.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:31.436+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:31.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:31.436+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:31.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:31.436+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:31.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:31.437+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:31.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:31.443+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:31.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:31.443+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:31.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:31.443+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:31.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:31.444+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:32.153 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:32.025+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4838 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:32.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:31.816+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:32.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:31.816+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:32.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:31.816+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:32.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:31.816+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:32.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:32.398+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:32.398+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:32.398+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:32.398+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:32.793 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:32.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:32.793 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:32.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:32.793 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:32.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:32.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:32.482+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:33.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:32.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:32.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:32.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:32.792+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:33.435+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:33.435+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:33.435+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:33.436+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:33.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:33.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:33.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:33.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:33.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:33.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:33.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:33.356+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:33.820+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:33.820+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:34.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:33.821+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:34.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:33.822+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:34.396+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:34.396+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:34.396+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:34.397+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:34.359+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:34.359+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:34.359+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:34.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:34.360+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:34.828+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:34.828+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:34.828+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:34.829+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:35.343+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:35.343+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:35.343+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:35.344+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:35.362+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:35.362+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:35.362+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:35.363+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:35.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:35.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:35.796+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:35.798+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:36.325+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:36.325+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:36.325+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:36.327+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:36.328+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:36.328+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:36.328+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:36.329+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:37.038 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:37.026+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 4947 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:37.038 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:36.777+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:37.038 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:36.777+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:37.038 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:36.777+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:37.038 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:36.778+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:37.306 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:37.285+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:37.306 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:37.285+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:37.306 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:37.285+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:37.306 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:37.286+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:37.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:37.303+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:37.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:37.303+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:37.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:37.303+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:37.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:37.304+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:38.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:37.763+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:37.763+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:37.763+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:37.764+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:38.320+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:38.320+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:38.320+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:38.321+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:38.246+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:38.246+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:38.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:38.246+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:38.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:38.246+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:38.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:38.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:38.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:38.758+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:39.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:39.328+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:39.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:39.328+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:39.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:39.328+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:39.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:39.328+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:39.209+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:39.209+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:39.209+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:39.209+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:40.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:39.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:40.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:39.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:40.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:39.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:40.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:39.715+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:40.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:40.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:40.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:40.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:40.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:40.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:40.687 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:40.362+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:40.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:40.223+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:40.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:40.223+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:40.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:40.223+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:40.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:40.223+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:40.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:40.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:40.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:40.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:40.686+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:41.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:41.384+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:41.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:41.384+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:41.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:41.384+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:41.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:41.385+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:41.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:41.267+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:41.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:41.267+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:41.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:41.267+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:41.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:41.268+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:41.725+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:41.725+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:41.725+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:41.726+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:42.340 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:42.027+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5041 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:42.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:42.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:42.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:42.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:42.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:42.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:42.340 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:42.241+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:42.338+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:42.338+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:42.338+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:42.339+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:42.702+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:42.702+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:42.702+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:42.702+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:43.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:43.378+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:43.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:43.378+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:43.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:43.378+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:43.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:43.378+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:43.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:43.251+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:43.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:43.251+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:43.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:43.251+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:43.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:43.251+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:43.726+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:43.726+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:43.726+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:43.727+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:44.526 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:44.422+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:44.527 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:44.422+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:44.527 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:44.422+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:44.527 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:44.423+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:44.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:44.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:44.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:44.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:44.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:44.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:44.527 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:44.249+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:44.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:44.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:44.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:44.728+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:45.274+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:45.274+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:45.274+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:45.274+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:45.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:45.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:45.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:45.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:45.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:45.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:45.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:45.442+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:45.738+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:45.738+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:45.738+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:45.739+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:46.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:46.476+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:46.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:46.476+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:46.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:46.476+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:46.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:46.477+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:46.315+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:46.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:46.315+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:46.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:46.315+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:46.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:46.315+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:47.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:46.728+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:47.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:46.728+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:47.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:46.728+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:47.030 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:46.729+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:47.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:47.028+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5139 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:47.600 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:47.484+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:47.600 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:47.484+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:47.600 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:47.484+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:47.600 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:47.484+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:47.600 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:47.336+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:47.600 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:47.336+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:47.600 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:47.336+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:47.600 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:47.336+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:47.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:47.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:47.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:47.760+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:48.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:48.503+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:48.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:48.503+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:48.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:48.503+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:48.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:48.504+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:48.312+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:48.312+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:48.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:48.312+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:48.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:48.312+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:48.739+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:49.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:48.739+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:49.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:48.739+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:49.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:48.740+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:49.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:49.297+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:49.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:49.297+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:49.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:49.297+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:49.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:49.297+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:49.709+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:49.709+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:49.709+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:49.709+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:49.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:49.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:49.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:49.545+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:50.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:50.547+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:50.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:50.547+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:50.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:50.547+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:50.667 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:50.548+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:50.667 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:50.347+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:50.667 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:50.347+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:50.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:50.347+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:50.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:50.347+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:50.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:50.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:50.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:50.665+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:51.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:51.591+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:51.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:51.591+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:51.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:51.591+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:51.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:51.593+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:51.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:51.376+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:51.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:51.376+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:51.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:51.376+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:51.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:51.377+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:51.650+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:51.650+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:51.650+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:51.651+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:52.369 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:52.029+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5237 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:52.642 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:52.367+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:52.642 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:52.367+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:52.642 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:52.367+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:52.642 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:52.367+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:52.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:52.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:52.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:52.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:52.684+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:52.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:52.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:52.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:52.641+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:53.403+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:53.403+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:53.403+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:53.403+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:53.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:53.634+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:53.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:53.634+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:53.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:53.634+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:53.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:53.635+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:53.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:53.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:53.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:53.708+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:54.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:54.439+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:54.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:54.439+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:54.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:54.439+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:54.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:54.440+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:54.695 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:54.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:54.695 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:54.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:54.695 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:54.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:54.695 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:54.622+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:54.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:54.692+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:54.692+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:54.692+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:54.693+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:55.728+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:55.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:55.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:55.730+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:55.587+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:55.587+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:55.587+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:55.588+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:55.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:55.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:55.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:55.486+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:56.711+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:56.711+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:56.711+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:56.712+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:56.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:56.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:56.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:56.542+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:56.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:56.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:56.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:56.535+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:57.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:56:57.030+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5351 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:56:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:57.553+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:57.553+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:57.553+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:57.554+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:57.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:57.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:57.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:57.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:57.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:57.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:57.545+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:57.670+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:57.670+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:57.670+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:57.671+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:58.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:58.688+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:58.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:58.688+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:58.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:58.688+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:58.900 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:58.688+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:58.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:58.541+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:58.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:58.541+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:58.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:58.541+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:58.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:58.542+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:56:58.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:58.498+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:58.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:58.498+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:58.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:58.498+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:58.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:58.499+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:59.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:59.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:59.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:59.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:59.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:59.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:56:59.523 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:56:59.486+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:59.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:59.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:59.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:56:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:56:59.642+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:56:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:59.520+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:59.520+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:59.520+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:56:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:56:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:56:59.522+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:00.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:00.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:00.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:00.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:00.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:00.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:00.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:00.666+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:00.440+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:00.440+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:00.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:00.440+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:00.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:00.441+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:00.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:00.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:00.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:00.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:00.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:00.486+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:00.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:00.488+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:01.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:01.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:01.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:01.679+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:01.513+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:01.513+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:01.513+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:01.514+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:01.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:01.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:01.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:01.459+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:02.447 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:02.030+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5458 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:02.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:02.658+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:02.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:02.658+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:02.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:02.658+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:02.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:02.658+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:02.445+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:02.445+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:02.445+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:02.445+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:02.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:02.506+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:02.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:02.506+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:02.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:02.506+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:02.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:02.507+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:03.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:03.468+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:03.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:03.468+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:03.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:03.468+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:03.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:03.469+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:03.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:03.414+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:03.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:03.414+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:03.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:03.414+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:03.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:03.415+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:03.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:03.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:03.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:03.683+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:04.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:04.428+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:04.688 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:04.428+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:04.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:04.428+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:04.689 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:04.428+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:04.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:04.416+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:04.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:04.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:04.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:04.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:04.690 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:04.417+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:04.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:04.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:04.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:04.687+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:05.686 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:05.380+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:05.686 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:05.380+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:05.686 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:05.380+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:05.686 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:05.381+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:05.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:05.404+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:05.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:05.404+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:05.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:05.404+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:05.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:05.404+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:05.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:05.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:05.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:05.685+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:06.651 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:06.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:06.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:06.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:06.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:06.350+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:06.652 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:06.350+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:06.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:06.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:06.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:06.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:06.652 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:06.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:06.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:06.452+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:06.649+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:06.649+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:06.649+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:06.650+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:07.338 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:07.031+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5554 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:07.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:07.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:07.653 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:07.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:07.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:07.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:07.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:07.417+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:07.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:07.336+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:07.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:07.336+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:07.654 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:07.336+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:07.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:07.337+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:07.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:07.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:07.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:07.652+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:08.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:08.311+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:08.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:08.311+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:08.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:08.311+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:08.616 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:08.311+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:08.617 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:08.435+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:08.617 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:08.435+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:08.617 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:08.435+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:08.617 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:08.436+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:08.614+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:08.614+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:08.614+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:08.615+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:09.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:09.328+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:09.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:09.328+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:09.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:09.328+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:09.607 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:09.329+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:09.607 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:09.471+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:09.607 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:09.471+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:09.607 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:09.471+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:09.607 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:09.472+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:09.604+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:09.604+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:09.604+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:09.605+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:10.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:10.319+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:10.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:10.319+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:10.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:10.319+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:10.585 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:10.320+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:10.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:10.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:10.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:10.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:10.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:10.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:10.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:10.462+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:10.582+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:10.582+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:10.582+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:10.584+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:11.624 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:11.336+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:11.624 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:11.336+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:11.625 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:11.336+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:11.625 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:11.337+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:11.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:11.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:11.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:11.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:11.626 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:11.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:11.626 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:11.483+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:11.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:11.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:11.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:11.623+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:12.382 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:12.032+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5650 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:12.662 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:12.380+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:12.662 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:12.380+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:12.662 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:12.380+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:12.662 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:12.381+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:12.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:12.437+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:12.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:12.437+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:12.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:12.437+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:12.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:12.437+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:12.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:12.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:12.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:12.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:12.661+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:13.408+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:13.408+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:13.408+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:13.409+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:13.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:13.476+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:13.476+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:13.476+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:13.477+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:13.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:13.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:13.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:13.700+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:14.456+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:14.456+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:14.456+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:14.456+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:14.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:14.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:14.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:14.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:14.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:14.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:14.402+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:14.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:14.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:14.699+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:14.700+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:15.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:15.400+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:15.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:15.400+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:15.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:15.400+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:15.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:15.401+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:15.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:15.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:15.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:15.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:15.486+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:16.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:15.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:15.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:15.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:15.730+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:16.367+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:16.367+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:16.367+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:16.368+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:16.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:16.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:16.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:16.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:16.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:16.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:16.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:16.502+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:17.035 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:16.732+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:17.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:16.732+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:17.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:16.732+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:17.036 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:16.734+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:17.404 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:17.033+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5747 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:17.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:17.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:17.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:17.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:17.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:17.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:17.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:17.403+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:17.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:17.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:17.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:17.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:17.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:17.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:17.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:17.462+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:18.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:17.733+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:17.733+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:17.733+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:17.734+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:18.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:18.450+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:18.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:18.450+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:18.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:18.450+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:18.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:18.450+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:18.771 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:18.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:18.771 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:18.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:18.771 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:18.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:18.771 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:18.483+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:18.768+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:19.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:18.768+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:19.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:18.768+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:19.203 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:18.769+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:19.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:19.476+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:19.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:19.476+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:19.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:19.476+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:19.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:19.477+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:19.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:19.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:19.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:19.483+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:19.786+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:19.786+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:19.786+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:19.788+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:20.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:20.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:20.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:20.435+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:20.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:20.510+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:20.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:20.510+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:20.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:20.510+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:20.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:20.511+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:20.800+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:20.800+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:20.800+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:20.800+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:21.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:21.390+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:21.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:21.390+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:21.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:21.390+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:21.655 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:21.392+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:21.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:21.505+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:21.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:21.505+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:21.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:21.505+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:21.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:21.506+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:21.809+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:21.809+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:21.809+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:21.810+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:22.351 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:22.034+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5842 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:22.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:22.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:22.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:22.350+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:22.463+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:22.463+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:22.463+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:22.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:22.464+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:22.763+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:22.763+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:22.763+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:22.764+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:23.486 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:23.317+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:23.486 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:23.317+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:23.486 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:23.317+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:23.486 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:23.317+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:23.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:23.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:23.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:23.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:23.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:23.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:23.789 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:23.485+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:23.786+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:23.786+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:23.786+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:23.787+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:24.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:24.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:24.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:24.350+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:24.515+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:24.515+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:24.515+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:24.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:24.516+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:24.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:24.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:24.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:24.792+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:25.565+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:25.565+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:25.565+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:25.565+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:25.333+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:25.333+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:25.333+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:25.334+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:25.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:25.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:25.825+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:25.826+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:26.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:26.557+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:26.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:26.557+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:26.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:26.557+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:26.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:26.558+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:26.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:26.381+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:26.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:26.381+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:26.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:26.381+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:26.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:26.382+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:27.162 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:27.035+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 5939 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:27.163 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:26.838+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:27.163 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:26.838+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:27.163 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:26.838+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:27.163 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:26.839+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:27.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:27.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:27.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:27.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:27.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:27.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:27.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:27.402+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:27.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:27.553+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:27.857 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:27.553+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:27.857 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:27.553+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:27.857 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:27.554+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:27.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:27.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:27.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:27.855+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:28.443+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:28.443+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:28.443+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:28.444+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:28.560+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:28.560+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:28.560+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:28.560+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:28.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:28.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:28.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:28.860+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:29.546 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:29.483+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:29.546 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:29.483+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:29.546 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:29.483+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:29.547 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:29.484+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:29.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:29.544+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:29.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:29.544+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:29.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:29.544+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:29.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:29.544+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:29.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:29.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:29.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:29.904+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:30.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:30.521+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:30.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:30.521+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:30.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:30.521+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:30.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:30.522+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:30.912 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:30.526+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:30.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:30.526+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:30.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:30.526+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:30.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:30.526+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:30.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:30.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:30.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:30.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:31.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:30.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:31.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:30.910+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:31.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:31.549+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:31.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:31.549+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:31.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:31.549+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:31.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:31.550+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:31.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:31.517+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:31.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:31.517+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:31.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:31.517+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:31.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:31.518+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:32.160 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:32.036+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6034 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:32.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:31.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:32.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:31.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:32.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:31.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:32.161 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:31.885+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:32.880 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:32.496+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:32.880 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:32.496+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:32.880 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:32.496+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:32.880 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:32.497+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:32.880 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:32.561+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:32.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:32.561+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:32.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:32.561+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:32.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:32.562+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:33.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:32.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:32.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:32.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:32.878+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:33.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:33.567+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:33.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:33.567+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:33.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:33.567+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:33.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:33.568+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:33.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:33.465+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:33.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:33.465+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:33.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:33.465+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:33.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:33.466+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:34.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:33.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:33.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:33.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:33.919+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:34.890 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:34.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:34.890 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:34.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:34.890 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:34.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:34.890 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:34.593+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:34.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:34.466+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:34.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:34.466+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:34.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:34.466+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:34.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:34.467+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:34.887+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:34.887+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:34.887+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:34.888+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:35.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:35.479+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:35.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:35.479+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:35.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:35.479+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:35.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:35.479+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:35.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:35.633+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:35.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:35.633+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:35.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:35.633+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:35.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:35.634+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:36.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:35.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:35.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:35.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:35.926+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:36.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:36.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:36.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:36.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:36.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:36.632+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:36.751 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:36.632+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:36.751 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:36.492+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:36.751 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:36.492+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:36.751 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:36.492+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:36.752 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:36.492+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:37.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:36.899+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:37.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:36.899+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:37.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:36.899+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:37.039 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:36.899+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:37.449 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:37.037+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6131 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:37.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:37.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:37.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:37.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:37.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:37.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:37.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:37.619+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:37.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:37.527+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:37.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:37.527+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:37.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:37.527+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:37.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:37.528+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:38.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:37.874+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:37.874+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:37.874+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:37.876+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:38.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:38.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:38.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:38.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:38.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:38.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:38.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:38.662+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:38.886 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:38.506+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:38.886 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:38.506+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:38.886 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:38.506+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:38.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:38.506+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:38.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:38.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:38.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:38.885+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:39.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:39.512+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:39.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:39.512+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:39.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:39.512+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:39.823 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:39.513+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:39.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:39.626+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:39.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:39.626+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:39.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:39.626+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:39.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:39.628+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:40.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:39.881+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:40.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:39.881+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:40.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:39.881+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:40.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:39.882+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:40.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:40.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:40.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:40.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:40.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:40.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:40.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:40.584+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:40.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:40.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:40.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:40.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:40.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:40.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:40.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:40.494+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:41.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:40.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:40.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:40.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:40.911+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:41.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:41.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:41.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:41.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:41.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:41.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:41.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:41.600+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:41.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:41.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:41.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:41.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:41.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:41.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:41.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:41.484+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:42.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:42.038+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6226 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:42.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:41.944+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:42.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:41.944+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:42.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:41.944+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:42.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:41.946+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:42.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:42.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:42.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:42.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:42.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:42.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:42.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:42.453+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:42.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:42.598+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:42.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:42.598+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:42.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:42.598+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:42.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:42.598+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:42.974+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:42.974+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:42.974+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:42.974+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:43.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:43.466+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:43.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:43.466+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:43.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:43.466+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:43.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:43.466+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:43.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:43.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:43.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:43.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:43.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:43.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:43.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:43.617+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:43.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:43.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:43.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:43.941+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:44.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:44.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:44.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:44.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:44.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:44.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:44.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:44.650+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:44.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:44.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:44.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:44.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:44.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:44.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:44.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:44.452+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:44.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:44.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:44.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:44.904+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:45.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:45.408+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:45.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:45.408+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:45.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:45.408+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:45.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:45.409+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:45.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:45.665+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:45.932 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:45.665+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:45.932 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:45.665+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:45.932 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:45.666+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:45.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:45.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:45.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:45.929+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:46.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:46.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:46.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:46.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:46.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:46.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:46.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:46.647+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:46.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:46.450+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:46.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:46.450+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:46.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:46.450+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:46.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:46.451+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:47.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:47.038+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6323 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:47.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:46.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:47.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:46.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:47.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:46.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:47.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:46.918+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:47.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:47.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:47.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:47.643+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:47.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:47.643+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:47.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:47.643+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:47.441+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:47.441+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:47.441+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:47.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:47.442+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:47.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:47.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:47.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:48.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:47.924+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:48.406+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:48.406+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:48.406+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:48.406+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:48.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:48.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:48.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:48.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:48.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:48.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:48.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:48.621+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:48.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:48.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:48.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:48.910+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:49.623 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:49.375+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:49.623 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:49.375+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:49.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:49.375+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:49.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:49.376+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:49.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:49.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:49.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:49.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:49.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:49.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:49.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:49.622+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:50.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:49.931+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:50.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:49.931+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:50.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:49.931+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:50.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:49.932+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:50.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:50.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:50.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:50.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:50.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:50.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:50.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:50.591+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:50.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:50.365+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:50.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:50.365+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:50.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:50.365+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:50.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:50.366+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:51.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:50.906+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:51.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:50.906+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:51.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:50.906+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:51.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:50.907+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:51.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:51.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:51.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:51.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:51.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:51.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:51.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:51.618+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:51.409+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:51.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:51.409+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:51.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:51.409+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:51.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:51.410+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:52.167 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:52.040+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6418 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:52.167 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:51.900+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:52.167 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:51.900+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:52.168 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:51.900+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:52.168 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:51.901+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:52.877 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:52.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:52.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:52.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:52.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:52.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:52.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:52.640+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:52.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:52.449+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:52.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:52.449+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:52.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:52.449+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:52.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:52.450+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:53.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:52.875+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:52.875+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:52.875+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:52.876+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:53.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:53.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:53.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:53.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:53.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:53.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:53.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:53.652+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:53.415+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:53.415+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:53.415+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:53.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:53.415+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:54.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:53.868+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:54.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:53.868+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:54.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:53.868+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:54.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:53.869+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:54.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:54.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:54.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:54.655+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:54.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:54.427+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:54.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:54.427+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:54.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:54.427+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:54.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:54.428+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:55.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:54.865+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:55.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:54.865+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:55.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:54.866+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:55.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:54.866+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:55.404+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:55.404+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:55.404+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:55.404+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:55.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:55.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:55.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:55.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:55.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:55.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:55.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:55.630+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:55.881 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:55.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:55.881 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:55.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:55.881 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:55.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:55.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:55.878+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:56.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:56.382+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:56.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:56.382+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:56.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:56.382+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:56.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:56.383+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:56.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:56.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:56.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:56.916+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:56.669+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:56.669+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:56.669+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:56.670+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:57.347 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:57:57.040+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6515 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:57:57.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:57.345+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:57.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:57.346+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:57.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:57.346+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:57.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:57.346+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:57.879+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:57.879+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:57.879+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:57.880+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:57.666+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:57.666+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:57.666+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:57.667+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:58.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:58.341+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:58.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:58.341+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:58.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:58.341+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:58.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:58.342+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:58.890+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:58.890+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:58.890+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:58.892+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:58.668+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:58.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:58.668+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:58.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:58.668+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:58.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:58.668+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:57:59.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:59.354+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:59.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:59.354+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:59.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:59.354+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:57:59.659 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:57:59.355+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:59.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:59.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:59.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:57:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:57:59.887+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:57:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:59.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:59.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:59.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:57:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:57:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:57:59.658+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:00.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:00.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:00.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:00.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:00.388+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:00.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:00.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:00.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:00.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:00.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:00.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:00.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:00.625+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:00.934+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:00.934+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:00.934+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:00.935+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:01.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:01.414+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:01.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:01.414+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:01.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:01.414+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:01.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:01.414+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:01.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:01.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:01.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:01.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:01.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:01.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:01.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:01.620+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:02.199 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:02.041+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6610 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:01.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:01.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:01.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:01.925+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:02.665 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:02.372+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:02.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:02.373+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:02.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:02.373+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:02.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:02.373+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:02.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:02.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:02.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:02.885+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:02.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:02.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:02.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:02.664+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:03.644 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:03.371+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:03.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:03.371+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:03.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:03.371+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:03.645 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:03.372+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:03.642+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:03.642+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:03.642+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:03.643+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:03.875+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:03.875+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:03.875+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:03.875+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:04.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:04.377+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:04.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:04.377+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:04.662 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:04.377+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:04.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:04.378+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:04.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:04.660+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:04.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:04.660+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:04.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:04.660+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:04.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:04.661+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:04.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:04.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:04.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:04.912+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:05.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:05.638+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:05.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:05.638+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:05.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:05.638+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:05.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:05.639+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:05.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:05.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:05.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:05.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:05.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:05.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:05.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:05.418+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:05.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:05.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:05.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:05.897+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:06.627+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:06.627+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:06.627+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:06.628+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:06.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:06.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:06.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:06.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:06.417+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:07.169 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:07.043+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6707 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:07.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:06.849+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:07.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:06.849+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:07.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:06.849+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:07.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:06.850+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:07.410+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:07.410+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:07.410+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:07.410+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:07.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:07.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:07.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:07.871+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:07.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:07.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:07.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:07.653+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:08.664 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:08.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:08.664 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:08.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:08.664 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:08.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:08.664 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:08.393+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:08.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:08.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:08.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:08.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:08.663+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:08.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:08.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:08.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:08.833+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:09.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:09.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:09.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:09.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:09.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:09.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:09.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:09.356+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:09.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:09.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:09.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:09.871+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:09.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:09.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:09.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:09.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:09.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:09.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:09.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:09.689+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:10.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:10.348+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:10.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:10.348+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:10.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:10.348+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:10.656 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:10.349+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:10.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:10.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:10.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:10.831+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:10.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:10.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:10.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:10.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:10.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:10.655+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:11.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:11.314+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:11.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:11.314+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:11.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:11.314+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:11.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:11.314+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:11.826+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:11.826+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:11.826+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:11.827+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:11.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:11.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:11.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:11.625+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:12.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:12.044+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6802 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:12.634 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:12.322+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:12.634 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:12.322+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:12.634 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:12.322+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:12.634 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:12.323+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:12.851+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:12.851+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:12.851+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:12.852+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:12.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:12.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:12.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:12.633+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:13.326+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:13.326+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:13.326+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:13.326+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:13.586+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:13.586+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:13.586+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:13.587+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:13.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:13.822+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:13.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:13.822+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:13.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:13.822+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:13.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:13.823+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:14.543+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:14.543+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:14.543+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:14.545+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:14.301+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:14.301+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:14.301+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:14.301+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:14.835+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:14.835+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:14.835+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:14.835+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:15.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:15.323+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:15.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:15.323+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:15.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:15.323+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:15.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:15.325+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:15.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:15.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:15.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:15.591+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:15.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:15.591+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:15.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:15.591+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:15.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:15.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:15.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:15.861+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:16.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:16.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:16.544+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:16.545+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:16.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:16.302+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:16.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:16.302+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:16.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:16.302+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:16.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:16.303+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:17.172 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:17.045+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6899 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:17.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:16.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:17.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:16.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:17.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:16.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:17.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:16.887+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:17.283+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:17.283+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:17.283+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:17.283+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:17.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:17.501+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:17.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:17.501+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:17.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:17.501+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:17.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:17.502+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:17.913+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:17.913+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:17.913+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:17.914+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:18.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:18.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:18.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:18.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:18.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:18.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:18.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:18.535+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:18.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:18.291+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:18.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:18.291+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:18.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:18.291+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:18.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:18.291+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:19.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:18.961+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:19.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:18.961+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:19.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:18.961+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:19.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:18.961+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:19.516 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:19.327+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.034424+0000 front 2023-12-19T18:47:47.034531+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:19.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:19.327+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.034514+0000 front 2023-12-19T18:47:47.034482+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:19.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:19.327+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.034454+0000 front 2023-12-19T18:47:47.034463+0000 (oldest deadline 2023-12-19T18:48:08.734290+0000) 2023-12-19T18:58:19.517 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:19.328+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:19.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:19.514+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:19.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:19.514+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:19.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:19.514+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:19.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:19.515+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:19.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:19.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:19.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:20.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:19.919+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:20.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:20.339+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:20.529+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:20.529+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:20.529+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:20.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:20.530+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:21.101 INFO:teuthology.orchestra.run.smithi059.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-19T18:58:21.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-19T18:58:21.098+0000 7f34132af700 0 monclient(hunting): authenticate timed out after 300 2023-12-19T18:58:21.385 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:20.954+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:21.385 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:20.954+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:21.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:20.954+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:21.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:20.955+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:21.661 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:21.383+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:21.661 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:21.488+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:21.661 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:21.488+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:21.661 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:21.488+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:21.661 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:21.489+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:21.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:58:21.837 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi059 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph orch ps'" 2023-12-19T18:58:21.916 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=63f517d953144809b3d10ecfa2be3948 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi059 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph orch ps'" 2023-12-19T18:58:21.919 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-12-19T18:58:21.931 INFO:tasks.kclient:Unmounting kernel clients... 2023-12-19T18:58:21.931 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:58:21.932 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T18:58:21.966 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T18:58:21.966 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T18:58:22.023 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T18:58:22.421 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:22.046+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 6994 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:22.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:21.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:22.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:21.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:22.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:21.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:22.422 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:21.951+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:22.419+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:22.495+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:22.495+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:22.495+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:22.496+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:22.912+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:22.912+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:22.912+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:22.913+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:23.459 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:23.390+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:23.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:23.457+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:23.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:23.457+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:23.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:23.457+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:23.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:23.458+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:23.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:23.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:23.910+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:23.910+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:24.385+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:24.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:24.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:24.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:24.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:24.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:24.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:24.417+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:24.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:24.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:24.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:24.939+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:25.359+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:25.414+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:25.414+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:25.414+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:25.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:25.414+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:25.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:25.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:25.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:26.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:25.967+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:26.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:26.391+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:26.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:26.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:26.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:26.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:26.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:26.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:26.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:26.379+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:27.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:27.047+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7091 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:27.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:26.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:27.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:26.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:27.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:26.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:27.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:26.942+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:27.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:27.420+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:27.362+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:27.362+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:27.362+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:27.362+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:28.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:27.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:28.404 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:27.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:28.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:27.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:28.405 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:27.959+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:28.459+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:28.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:28.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:28.402+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:28.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:28.403+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:29.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:28.977+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:29.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:28.977+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:29.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:28.977+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:29.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:28.978+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:29.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:29.462+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:29.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:29.443+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:29.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:29.443+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:29.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:29.443+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:29.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:29.444+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:29.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:29.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:29.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:29.939+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:30.937 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:30.458+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:30.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:30.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:30.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:30.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:30.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:30.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:30.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:30.482+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:30.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:30.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:30.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:30.935+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:31.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:31.416+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:31.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:31.492+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:31.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:31.492+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:31.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:31.492+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:31.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:31.493+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:32.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:32.048+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7186 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:31.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.754435+0000 front 2023-12-19T18:47:47.754377+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:31.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.754467+0000 front 2023-12-19T18:47:47.754367+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:31.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.754414+0000 front 2023-12-19T18:47:47.754421+0000 (oldest deadline 2023-12-19T18:48:13.054333+0000) 2023-12-19T18:58:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:31.940+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:32.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:32.427+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:32.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:32.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 since back 2023-12-19T18:47:47.424203+0000 front 2023-12-19T18:47:47.424154+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:32.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:32.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 since back 2023-12-19T18:47:47.424188+0000 front 2023-12-19T18:47:47.424164+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:32.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:32.482+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 since back 2023-12-19T18:47:47.424127+0000 front 2023-12-19T18:47:47.424094+0000 (oldest deadline 2023-12-19T18:48:13.324124+0000) 2023-12-19T18:58:32.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:32.483+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:33.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:32.920+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:33.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:33.514+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:33.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:33.458+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:34.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:33.888+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:34.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:34.545+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:34.447+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:35.118 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:34.862+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:35.908 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:35.580+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:35.908 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:35.471+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:36.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:35.906+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:36.614+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:36.444+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:37.172 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:37.049+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7283 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:37.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:36.896+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:37.403+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:37.891 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:37.632+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:38.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:37.889+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:38.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:38.386+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:38.844+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:38.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:38.659+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:39.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:39.428+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:39.854+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:39.676+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:40.638+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:40.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:40.398+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:41.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:40.877+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:41.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:41.384+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:41.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:41.384+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:41.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:41.384+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:41.594 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:41.384+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:41.839+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:41.591+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:42.415 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:42.050+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7378 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:42.548+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:42.413+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:42.413+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:42.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:42.413+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:42.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:42.413+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:42.817+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:43.579+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:43.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:43.435+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:43.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:43.436+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:43.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:43.436+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:43.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:43.436+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:43.838+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:44.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:44.577+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:44.430+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:44.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:44.430+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:44.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:44.430+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:44.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:44.431+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:44.809+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:45.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:45.583+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:45.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:45.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:45.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:45.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:45.393+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:45.840+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:46.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:46.596+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:46.434+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:46.434+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:46.435+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:46.435+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:47.054 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:46.795+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:47.407 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:47.051+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7475 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:47.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:47.573+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:47.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:47.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:47.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:47.478+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:47.828+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:48.844 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:48.564+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:48.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:48.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:48.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:48.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:48.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:48.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:48.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:48.488+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:48.842+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:49.558 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:49.456+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:49.558 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:49.456+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:49.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:49.456+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:49.559 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:49.457+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:49.794+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:49.556+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:50.758 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:50.517+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:50.758 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:50.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:50.758 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:50.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:50.758 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:50.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:50.758 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:50.484+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:51.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:50.756+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:51.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:51.526+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:51.444+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:51.444+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:51.444+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:51.445+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:52.055 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:51.733+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:52.440 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:52.052+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7570 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:52.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:52.575+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:52.438+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:52.438+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:52.438+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:52.438+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:53.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:52.763+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:53.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:53.441+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:53.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:53.441+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:53.550 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:53.441+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:53.551 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:53.442+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:53.779+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:53.590+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:54.635+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:54.635+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:54.635+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:54.635+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:54.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:54.430+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:54.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:54.430+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:54.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:54.430+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:54.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:54.430+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:54.818+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:54.818+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:54.818+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:54.818+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:55.863 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:55.449+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:55.863 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:55.449+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:55.863 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:55.449+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:55.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:55.450+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:55.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:55.587+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:55.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:55.587+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:55.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:55.587+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:55.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:55.588+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:55.861+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:55.861+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:55.861+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:55.862+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:56.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:56.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:56.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:56.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:56.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:56.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:56.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:56.497+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:56.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:56.588+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:56.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:56.588+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:56.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:56.588+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:56.901 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:56.589+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:57.178 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:58:57.053+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7667 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:58:57.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:56.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:57.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:56.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:57.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:56.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:57.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:56.898+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:57.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:57.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:57.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:57.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:57.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:57.594+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:57.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:57.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:57.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:57.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:57.518+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:57.946+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:57.946+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:57.946+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:57.947+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:58.581+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:58.581+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:58.581+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:58.582+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:58:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:58.542+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:58.542+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:58.542+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:58.542+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:59.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:58.982+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:59.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:58.982+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:59.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:58.982+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:58:59.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:58.983+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:59.558+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:59.558+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:59.558+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:58:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:58:59.558+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:58:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:59.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:59.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:59.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:58:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:58:59.571+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:59.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:59.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:59.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:58:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:58:59.953+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:00.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:00.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:00.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:00.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:00.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:00.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:00.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:00.560+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:00.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:00.514+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:00.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:00.514+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:00.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:00.514+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:00.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:00.515+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:00.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:00.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:00.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:00.926+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:01.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:01.515+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:01.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:01.515+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:01.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:01.515+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:01.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:01.515+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:01.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:01.518+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:01.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:01.518+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:01.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:01.518+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:01.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:01.519+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:02.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:02.054+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7762 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:02.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:01.932+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:02.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:01.932+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:02.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:01.932+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:02.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:01.933+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:02.766 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:02.555+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:02.766 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:02.555+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:02.767 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:02.555+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:02.767 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:02.556+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:02.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:02.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:02.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:02.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:02.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:02.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:02.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:02.501+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:02.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:02.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:02.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:02.963+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:03.507+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:03.507+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:03.507+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:03.507+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:03.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:03.522+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:03.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:03.522+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:03.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:03.522+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:03.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:03.523+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:03.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:03.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:03.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:03.957+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:04.484+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:04.484+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:04.484+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:04.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:04.485+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:04.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:04.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:04.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:04.511+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:05.002+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:05.002+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:05.002+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:05.003+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:05.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:05.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:05.838 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:05.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:05.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:05.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:05.839 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:05.536+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:05.472+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:05.472+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:05.472+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:05.839 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:05.473+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:06.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:06.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:06.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:06.046+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:06.449+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:06.449+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:06.449+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:06.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:06.450+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:06.525+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:06.525+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:06.525+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:06.526+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:07.424 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:07.055+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7859 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:07.424 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:07.057+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:07.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:07.057+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:07.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:07.057+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:07.425 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:07.060+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:07.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:07.422+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:07.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:07.422+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:07.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:07.422+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:07.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:07.423+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:07.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:07.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:07.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:07.492+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:08.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:08.064+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:08.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:08.064+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:08.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:08.064+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:08.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:08.065+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:08.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:08.388+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:08.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:08.388+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:08.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:08.388+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:08.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:08.389+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:08.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:08.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:08.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:08.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:08.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:08.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:08.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:08.498+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:09.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:09.078+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:09.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:09.078+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:09.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:09.078+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:09.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:09.079+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:09.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:09.415+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:09.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:09.415+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:09.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:09.415+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:09.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:09.416+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:09.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:09.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:09.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:09.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:09.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:09.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:09.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:09.452+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:10.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:10.051+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:10.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:10.051+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:10.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:10.051+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:10.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:10.052+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:10.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:10.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:10.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:10.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:10.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:10.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:10.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:10.501+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:10.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:10.435+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:10.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:10.435+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:10.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:10.435+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:10.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:10.437+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:11.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:11.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:11.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:11.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:11.097+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:11.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:11.457+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:11.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:11.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:11.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:11.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:11.459+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:11.487+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:11.487+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:11.487+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:11.487+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:12.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:12.056+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 7954 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:12.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:12.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:12.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:12.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:12.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:12.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:12.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:12.111+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:12.463+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:12.463+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:12.463+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:12.463+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:12.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:12.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:12.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:12.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:12.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:12.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:12.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:12.495+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:13.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:13.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:13.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:13.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:13.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:13.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:13.432 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:13.083+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:13.429+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:13.429+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:13.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:13.429+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:13.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:13.430+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:13.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:13.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:13.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:13.505+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:14.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:14.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:14.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:14.102+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:14.461+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:14.461+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:14.461+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:14.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:14.462+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:14.473+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:14.473+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:14.473+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:14.473+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:15.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:15.108+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:15.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:15.108+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:15.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:15.108+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:15.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:15.109+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:15.441+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:15.441+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:15.441+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:15.442+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:15.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:15.471+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:15.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:15.471+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:15.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:15.471+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:15.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:15.472+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:16.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:16.140+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:16.430 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:16.140+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:16.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:16.140+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:16.431 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:16.141+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:16.428+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:16.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:16.428+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:16.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:16.428+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:16.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:16.429+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:16.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:16.459+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:16.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:16.459+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:16.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:16.459+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:16.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:16.460+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:17.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:17.057+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8051 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:17.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:17.179+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:17.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:17.179+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:17.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:17.179+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:17.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:17.180+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:17.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:17.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:17.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:17.485+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:17.452+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:17.452+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:17.452+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:17.453+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:18.521 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:18.211+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:18.521 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:18.211+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:18.521 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:18.211+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:18.521 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:18.212+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:18.522 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:18.425+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:18.522 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:18.425+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:18.522 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:18.425+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:18.522 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:18.427+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:18.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:18.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:18.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:18.519+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:19.484 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:19.213+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:19.485 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:19.213+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:19.485 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:19.213+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:19.485 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:19.215+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:19.485 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:19.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:19.485 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:19.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:19.485 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:19.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:19.486 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:19.430+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:19.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:19.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:19.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:19.483+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:20.191+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:20.191+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:20.191+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:20.191+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:20.413+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:20.413+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:20.413+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:20.414+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:20.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:20.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:20.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:20.504+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:21.488 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:21.205+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:21.488 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:21.205+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:21.488 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:21.205+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:21.488 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:21.206+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:21.488 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:21.394+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:21.489 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:21.394+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:21.489 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:21.394+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:21.489 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:21.395+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:21.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:21.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:21.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:21.486+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:22.420 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:22.058+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8146 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:22.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:22.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:22.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:22.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:22.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:22.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:22.421 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:22.179+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:22.418+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:22.418+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:22.418+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:22.419+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:22.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:22.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:22.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:22.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:22.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:22.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:22.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:22.521+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:23.193+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:23.193+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:23.193+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:23.194+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:23.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:23.381+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:23.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:23.381+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:23.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:23.381+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:23.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:23.381+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:23.495+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:23.495+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:23.495+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:23.495+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:24.194+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:24.194+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:24.194+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:24.195+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:24.367+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:24.367+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:24.367+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:24.368+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:24.454+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:24.454+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:24.454+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:24.454+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:25.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:25.169+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:25.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:25.169+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:25.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:25.169+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:25.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:25.170+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:25.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:25.398+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:25.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:25.398+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:25.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:25.398+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:25.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:25.398+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:25.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:25.407+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:25.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:25.407+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:25.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:25.407+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:25.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:25.408+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:26.174+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:26.174+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:26.174+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:26.175+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:26.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:26.422+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:26.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:26.422+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:26.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:26.422+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:26.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:26.423+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:26.455+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:26.455+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:26.455+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:26.456+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:27.391 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:27.059+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8243 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:27.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:27.175+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:27.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:27.175+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:27.391 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:27.175+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:27.392 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:27.176+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:27.389+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:27.389+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:27.389+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:27.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:27.389+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:27.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:27.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:27.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:27.493+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:28.384+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:28.384+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:28.384+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:28.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:28.385+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:28.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:28.155+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:28.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:28.155+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:28.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:28.155+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:28.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:28.156+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:28.465+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:28.465+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:28.465+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:28.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:28.466+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:29.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:29.185+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:29.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:29.185+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:29.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:29.185+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:29.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:29.186+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:29.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:29.418+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:29.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:29.419+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:29.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:29.419+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:29.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:29.419+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:29.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:29.342+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:29.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:29.342+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:29.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:29.342+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:29.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:29.343+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:30.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:30.158+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:30.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:30.158+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:30.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:30.158+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:30.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:30.159+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:30.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:30.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:30.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:30.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:30.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:30.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:30.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:30.394+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:30.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:30.351+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:30.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:30.351+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:30.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:30.351+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:30.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:30.352+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:31.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:31.149+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:31.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:31.149+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:31.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:31.149+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:31.415 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:31.150+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:31.416 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:31.308+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:31.416 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:31.308+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:31.416 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:31.308+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:31.416 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:31.309+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:31.665 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:31.413+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:31.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:31.413+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:31.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:31.413+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:31.666 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:31.413+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:32.356 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:32.061+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8338 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:32.356 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:32.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:32.356 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:32.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:32.356 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:32.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:32.356 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:32.101+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:32.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:32.353+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:32.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:32.353+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:32.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:32.353+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:32.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:32.355+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:32.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:32.455+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:32.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:32.455+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:32.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:32.455+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:32.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:32.455+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:33.389 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:33.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:33.389 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:33.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:33.389 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:33.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:33.389 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:33.111+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:33.387+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:33.387+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:33.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:33.387+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:33.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:33.388+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:33.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:33.498+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:33.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:33.498+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:33.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:33.498+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:33.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:33.498+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:34.146+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:34.146+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:34.146+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:34.146+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:34.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:34.373+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:34.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:34.373+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:34.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:34.373+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:34.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:34.374+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:34.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:34.523+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:34.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:34.523+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:34.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:34.523+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:34.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:34.524+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:35.151+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:35.151+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:35.151+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:35.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:35.152+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:35.364+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:35.364+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:35.364+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:35.365+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:35.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:35.488+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:35.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:35.488+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:35.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:35.488+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:35.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:35.489+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:36.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:36.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:36.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:36.122+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:36.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:36.360+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:36.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:36.360+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:36.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:36.360+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:36.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:36.360+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:36.475+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:36.475+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:36.475+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:36.476+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:37.328 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:37.062+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8435 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:37.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:37.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:37.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:37.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:37.328 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:37.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:37.329 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:37.124+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:37.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:37.326+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:37.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:37.326+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:37.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:37.326+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:37.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:37.326+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:37.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:37.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:37.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:37.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:37.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:37.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:37.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:37.521+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:38.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:38.145+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:38.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:38.145+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:38.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:38.145+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:38.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:38.145+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:38.277+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:38.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:38.277+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:38.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:38.277+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:38.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:38.278+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:38.495+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:38.495+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:38.495+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:38.496+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:39.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:39.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:39.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:39.123+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:39.283+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:39.283+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:39.283+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:39.284+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:39.450+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:39.450+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:39.450+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:39.450+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:40.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:40.146+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:40.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:40.146+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:40.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:40.146+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:40.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:40.146+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:40.291+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:40.291+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:40.291+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:40.292+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:40.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:40.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:40.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:40.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:40.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:40.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:40.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:40.426+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:41.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:41.255+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:41.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:41.255+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:41.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:41.255+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:41.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:41.256+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:41.441 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:41.129+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:41.442 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:41.129+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:41.442 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:41.129+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:41.442 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:41.130+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:41.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:41.438+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:41.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:41.438+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:41.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:41.438+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:41.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:41.439+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:42.419 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:42.232+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:42.420 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:42.232+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:42.420 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:42.232+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:42.420 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:42.233+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:42.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:42.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:42.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:42.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:42.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:42.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:42.420 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:42.158+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:42.421 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:42.063+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8530 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:42.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:42.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:42.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:42.418+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:43.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:43.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:43.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:43.158+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:43.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:43.244+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:43.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:43.244+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:43.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:43.244+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:43.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:43.246+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:43.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:43.370+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:43.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:43.370+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:43.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:43.370+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:43.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:43.371+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:44.137+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:44.137+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:44.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:44.137+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:44.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:44.138+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:44.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:44.266+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:44.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:44.266+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:44.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:44.266+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:44.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:44.267+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:44.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:44.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:44.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:44.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:44.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:44.355+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:44.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:44.356+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:45.317+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:45.317+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:45.317+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:45.318+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:45.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:45.132+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:45.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:45.132+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:45.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:45.132+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:45.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:45.132+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:45.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:45.254+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:45.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:45.254+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:45.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:45.254+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:45.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:45.256+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:46.174+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:46.174+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:46.174+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:46.175+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:46.264+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:46.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:46.264+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:46.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:46.264+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:46.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:46.265+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:46.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:46.299+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:46.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:46.299+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:46.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:46.299+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:46.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:46.300+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:47.343 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:47.064+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8627 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:47.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:47.200+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:47.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:47.200+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:47.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:47.200+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:47.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:47.200+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:47.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:47.297+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:47.344 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:47.297+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:47.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:47.297+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:47.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:47.298+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:47.341+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:47.341+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:47.341+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:47.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:47.342+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:48.181+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:48.181+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:48.181+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:48.182+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:48.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:48.339+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:48.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:48.339+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:48.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:48.339+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:48.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:48.340+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:48.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:48.350+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:48.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:48.350+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:48.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:48.350+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:48.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:48.350+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:49.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:49.134+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:49.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:49.134+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:49.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:49.134+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:49.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:49.134+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:49.387 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:49.383+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:49.387 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:49.383+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:49.387 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:49.383+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:49.387 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:49.384+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:49.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:49.397+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:49.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:49.397+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:49.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:49.397+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:49.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:49.397+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:50.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:50.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:50.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:50.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:50.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:50.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:50.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:50.346+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:50.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:50.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:50.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:50.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:50.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:50.165+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:50.419 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:50.166+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:50.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:50.416+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:50.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:50.416+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:50.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:50.416+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:50.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:50.416+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:51.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:51.137+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:51.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:51.137+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:51.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:51.137+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:51.428 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:51.138+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:51.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:51.298+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:51.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:51.298+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:51.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:51.299+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:51.429 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:51.299+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:51.425+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:51.425+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:51.425+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:51.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:51.426+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:52.190 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:52.065+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8722 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:52.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:52.163+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:52.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:52.163+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:52.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:52.163+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:52.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:52.164+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:52.252+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:52.252+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:52.252+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:52.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:52.253+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:52.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:52.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:52.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:52.451+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:52.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:52.452+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:53.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:53.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:53.178+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:53.179+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:53.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:53.291+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:53.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:53.291+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:53.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:53.291+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:53.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:53.292+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:53.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:53.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:53.493+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:53.494+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:54.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:54.217+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:54.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:54.217+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:54.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:54.217+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:54.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:54.217+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:54.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:54.342+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:54.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:54.342+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:54.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:54.342+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:54.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:54.343+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:54.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:54.447+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:54.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:54.447+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:54.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:54.447+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:54.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:54.447+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:55.362 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:55.208+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:55.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:55.208+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:55.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:55.208+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:55.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:55.209+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:55.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:55.360+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:55.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:55.360+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:55.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:55.360+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:55.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:55.361+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:55.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:55.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:55.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:55.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:55.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:55.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:55.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:55.495+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:56.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:56.234+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:56.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:56.234+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:56.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:56.234+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:56.514 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:56.235+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:56.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:56.403+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:56.514 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:56.403+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:56.515 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:56.403+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:56.515 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:56.404+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:56.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:56.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:56.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:56.512+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:57.386 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:57.266+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:57.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:57.266+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:57.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:57.266+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:57.387 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:57.266+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:57.387 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T18:59:57.066+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8819 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T18:59:57.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:57.549+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:57.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:57.549+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:57.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:57.549+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:57.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:57.550+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:57.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:57.384+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:57.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:57.384+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:57.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:57.384+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:57.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:57.385+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:58.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:58.262+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:58.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:58.262+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:58.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:58.262+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:58.363 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:58.264+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:58.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:58.573+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:58.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:58.573+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:58.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:58.573+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:58.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:58.574+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:58.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:58.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:58.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:58.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:58.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:58.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:58.703 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:58.361+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:59.537 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:59.295+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:59.538 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:59.295+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:59.538 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:59.295+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T18:59:59.538 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T18:59:59.296+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T18:59:59.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:59.354+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:59.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:59.354+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:59.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:59.354+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T18:59:59.539 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T18:59:59.354+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T18:59:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:59.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:59.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:59.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T18:59:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 18:59:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T18:59:59.535+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:00.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:00.311+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:00.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:00.311+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:00.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:00.311+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:00.565 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:00.311+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:00.565 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:00.350+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:00.565 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:00.350+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:00.565 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:00.350+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:00.565 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:00.351+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:00.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:00.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:00.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:00.563+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:01.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:01.546+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:01.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:01.546+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:01.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:01.546+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:01.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:01.547+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:01.668 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:01.306+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:01.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:01.306+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:01.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:01.306+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:01.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:01.307+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:01.669 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:01.322+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:01.669 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:01.322+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:01.669 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:01.322+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:01.670 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:01.323+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:02.335 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:02.067+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 8914 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:02.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:02.333+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:02.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:02.333+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:02.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:02.333+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:02.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:02.334+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:02.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:02.334+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:02.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:02.334+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:02.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:02.334+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:02.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:02.335+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:02.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:02.528+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:02.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:02.528+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:02.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:02.528+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:02.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:02.529+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:03.567 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:03.345+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:03.567 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:03.345+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:03.567 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:03.345+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:03.567 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:03.346+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:03.568 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:03.292+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:03.568 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:03.292+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:03.568 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:03.292+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:03.568 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:03.292+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:03.568 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:03.486+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:03.568 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:03.486+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:03.569 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:03.486+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:03.569 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:03.487+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:04.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:04.394+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:04.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:04.395+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:04.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:04.395+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:04.494 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:04.395+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:04.494 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:04.275+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:04.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:04.275+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:04.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:04.275+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:04.495 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:04.276+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:04.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:04.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:04.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:04.492+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:05.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:05.430+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:05.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:05.430+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:05.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:05.430+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:05.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:05.431+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:05.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:05.285+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:05.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:05.285+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:05.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:05.285+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:05.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:05.286+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:05.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:05.514+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:05.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:05.514+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:05.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:05.514+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:05.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:05.515+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:06.316+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:06.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:06.316+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:06.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:06.316+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:06.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:06.317+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:06.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:06.434+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:06.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:06.434+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:06.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:06.434+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:06.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:06.435+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:06.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:06.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:06.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:06.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:06.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:06.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:06.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:06.487+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:07.324 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:07.068+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9011 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:07.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:07.523+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:07.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:07.523+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:07.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:07.523+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:07.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:07.523+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:07.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:07.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:07.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:07.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:07.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:07.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:07.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:07.445+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:07.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:07.322+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:07.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:07.322+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:07.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:07.322+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:07.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:07.323+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:08.555+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:08.555+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:08.555+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:08.555+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:08.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:08.422+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:08.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:08.422+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:08.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:08.422+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:08.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:08.423+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:08.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:08.359+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:08.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:08.359+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:08.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:08.359+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:08.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:08.359+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:09.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:09.402+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:09.563 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:09.402+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:09.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:09.402+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:09.564 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:09.403+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:09.564 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:09.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:09.564 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:09.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:09.564 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:09.344+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:09.564 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:09.345+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:09.561+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:09.561+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:09.561+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:09.561+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:10.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:10.330+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:10.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:10.330+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:10.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:10.330+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:10.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:10.330+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:10.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:10.376+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:10.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:10.376+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:10.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:10.376+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:10.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:10.377+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:10.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:10.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:10.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:10.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:10.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:10.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:10.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:10.536+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:11.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:11.426+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:11.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:11.426+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:11.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:11.426+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:11.669 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:11.427+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:11.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:11.541+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:11.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:11.541+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:11.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:11.541+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:11.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:11.541+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:11.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:11.333+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:11.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:11.333+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:11.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:11.333+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:11.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:11.334+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:12.380 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:12.070+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9106 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:12.380 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:12.286+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:12.380 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:12.286+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:12.380 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:12.286+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:12.380 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:12.287+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:12.378+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:12.378+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:12.378+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:12.379+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:12.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:12.524+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:12.524+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:12.524+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:12.524+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:13.568 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:13.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:13.568 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:13.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:13.568 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:13.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:13.569 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:13.492+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:13.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:13.258+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:13.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:13.258+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:13.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:13.258+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:13.569 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:13.259+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:13.570 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:13.423+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:13.570 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:13.423+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:13.570 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:13.423+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:13.570 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:13.424+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:14.244+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:14.244+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:14.244+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:14.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:14.245+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:14.461+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:14.461+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:14.461+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:14.461+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:14.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:14.421+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:14.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:14.421+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:14.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:14.421+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:14.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:14.422+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:15.414+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:15.414+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:15.414+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:15.414+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:15.442+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:15.442+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:15.442+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:15.443+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:15.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:15.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:15.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:15.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:15.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:15.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:15.702 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:15.203+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:16.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:16.454+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:16.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:16.454+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:16.495 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:16.454+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:16.496 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:16.455+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:16.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:16.242+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:16.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:16.242+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:16.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:16.242+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:16.496 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:16.243+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:16.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:16.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:16.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:16.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:16.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:16.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:16.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:16.449+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:17.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:17.071+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9203 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:17.242+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:17.242+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:17.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:17.242+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:17.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:17.243+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:17.494+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:17.494+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:17.494+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:17.494+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:17.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:17.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:17.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:17.463+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:18.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:18.435+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:18.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:18.435+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:18.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:18.435+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:18.473 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:18.436+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:18.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:18.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:18.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:18.474 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:18.203+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:18.471+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:18.471+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:18.471+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:18.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:18.472+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:19.182+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:19.182+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:19.182+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:19.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:19.183+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:19.460+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:19.460+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:19.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:19.460+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:19.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:19.461+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:19.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:19.449+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:19.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:19.449+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:19.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:19.449+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:19.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:19.450+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:20.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:20.170+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:20.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:20.170+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:20.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:20.170+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:20.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:20.171+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:20.468+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:20.468+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:20.468+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:20.469+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:20.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:20.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:20.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:20.460+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:21.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:21.190+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:21.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:21.190+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:21.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:21.190+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:21.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:21.190+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:21.471+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:21.471+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:21.471+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:21.472+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:21.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:21.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:21.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:21.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:21.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:21.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:21.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:21.469+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-19T19:00:22.108 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2023-12-19T19:00:22.108 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T19:00:22.108 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:00:22.127 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-19T19:00:22.127 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:00:22.183 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T19:00:22.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:22.072+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9295 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:22.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:22.233+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:22.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:22.233+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:22.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:22.233+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:22.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:22.234+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:22.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:22.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:22.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:22.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:22.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:22.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:22.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:22.488+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:22.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:22.476+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:22.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:22.476+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:22.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:22.476+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:22.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:22.477+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:23.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:23.515+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:23.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:23.515+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:23.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:23.515+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:23.569 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:23.516+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:23.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:23.282+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:23.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:23.282+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:23.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:23.282+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:23.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:23.283+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:23.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:23.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:23.570 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:23.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:23.571 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:23.483+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:23.571 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:23.483+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:24.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:24.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:24.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:24.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:24.483+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:24.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:24.310+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:24.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:24.310+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:24.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:24.311+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:24.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:24.311+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:24.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:24.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:24.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:24.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:24.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:24.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:24.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:24.554+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:25.576 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:25.464+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:25.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:25.464+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:25.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:25.464+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:25.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:25.464+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:25.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:25.264+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:25.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:25.264+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:25.578 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:25.264+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:25.578 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:25.265+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:25.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:25.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:25.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:25.575+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:26.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:26.312+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:26.592 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:26.312+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:26.593 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:26.312+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:26.593 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:26.313+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:26.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:26.475+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:26.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:26.475+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:26.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:26.475+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:26.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:26.476+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:26.590+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:26.590+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:26.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:26.590+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:26.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:26.591+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:27.439 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:27.073+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9389 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:27.439 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:27.275+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:27.439 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:27.275+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:27.440 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:27.275+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:27.440 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:27.276+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:27.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:27.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:27.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:27.571+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:27.437+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:27.437+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:27.437+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:27.438+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:28.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:28.283+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:28.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:28.283+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:28.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:28.283+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:28.577 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:28.284+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:28.577 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:28.443+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:28.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:28.443+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:28.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:28.443+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:28.578 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:28.443+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:28.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:28.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:28.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:28.575+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:29.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:29.237+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:29.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:29.237+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:29.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:29.237+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:29.538 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:29.238+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:29.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:29.480+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:29.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:29.480+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:29.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:29.480+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:29.539 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:29.480+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:29.536+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:29.536+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:29.536+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:29.537+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:30.188+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:30.188+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:30.188+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:30.188+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:30.510+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:30.510+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:30.510+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:30.510+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:30.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:30.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:30.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:30.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:30.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:30.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:30.540+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:31.503 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:31.229+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:31.503 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:31.229+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:31.503 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:31.229+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:31.504 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:31.230+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:31.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:31.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:31.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:31.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:31.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:31.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:31.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:31.500+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:31.855 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:31.536+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:31.855 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:31.536+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:31.855 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:31.536+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:31.855 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:31.536+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:32.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:32.073+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9480 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:32.479 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:32.220+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:32.479 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:32.220+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:32.479 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:32.220+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:32.479 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:32.221+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:32.577+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:32.577+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:32.577+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:32.578+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:32.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:32.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:32.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:32.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:32.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:32.477+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:32.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:32.477+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:33.184+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:33.184+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:33.184+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:33.184+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:33.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:33.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:33.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:33.581+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:33.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:33.505+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:33.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:33.505+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:33.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:33.505+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:33.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:33.506+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:34.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:34.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:34.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:34.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:34.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:34.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:34.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:34.161+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:34.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:34.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:34.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:34.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:34.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:34.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:34.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:34.463+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:34.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:34.614+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:34.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:34.614+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:34.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:34.614+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:34.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:34.615+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:35.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:35.196+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:35.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:35.196+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:35.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:35.196+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:35.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:35.197+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:35.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:35.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:35.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:35.575+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:35.455+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:35.455+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:35.455+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:35.456+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:36.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:36.238+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:36.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:36.238+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:36.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:36.238+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:36.508 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:36.239+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:36.587+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:36.587+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:36.587+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:36.588+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:36.505+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:36.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:36.505+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:36.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:36.505+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:36.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:36.506+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:37.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:37.075+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9569 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:37.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:37.229+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:37.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:37.229+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:37.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:37.229+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:37.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:37.231+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:37.619+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:37.619+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:37.619+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:37.620+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:37.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:37.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:37.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:37.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:37.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:37.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:37.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:37.545+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:38.550 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:38.267+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:38.551 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:38.267+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:38.551 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:38.267+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:38.551 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:38.268+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:38.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:38.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:38.668+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:38.668+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:38.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:38.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:38.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:38.549+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:39.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:39.269+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:39.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:39.269+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:39.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:39.269+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:39.549 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:39.270+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:39.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:39.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:39.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:39.696+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:39.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:39.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:39.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:39.545+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:39.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:39.546+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:40.594 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:40.294+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:40.594 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:40.294+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:40.594 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:40.294+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:40.595 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:40.294+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:40.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:40.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:40.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:40.685+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:40.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:40.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:40.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:40.592+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:41.605 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:41.342+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:41.605 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:41.342+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:41.605 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:41.342+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:41.606 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:41.342+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:41.603+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:41.603+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:41.603+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:41.604+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:41.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:41.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:41.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:41.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:41.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:41.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:41.724+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:42.369 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:42.077+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9657 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:42.367+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:42.367+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:42.367+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:42.368+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:42.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:42.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:42.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:42.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:42.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:42.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:42.584+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:42.761+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:42.761+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:42.761+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:42.762+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:43.329+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:43.329+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:43.329+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:43.331+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:43.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:43.563+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:43.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:43.563+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:43.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:43.563+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:43.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:43.563+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:44.142 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:43.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:44.142 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:43.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:44.142 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:43.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:44.143 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:43.717+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:44.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:44.375+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:44.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:44.375+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:44.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:44.375+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:44.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:44.376+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:44.750 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:44.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:44.750 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:44.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:44.750 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:44.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:44.750 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:44.562+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:44.747+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:44.747+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:44.747+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:44.748+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:45.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:45.343+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:45.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:45.343+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:45.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:45.343+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:45.609 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:45.344+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:45.758+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:45.758+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:45.758+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:45.759+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:45.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:45.606+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:45.606+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:45.606+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:45.607+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:46.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:46.369+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:46.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:46.369+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:46.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:46.369+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:46.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:46.370+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:46.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:46.631+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:46.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:46.631+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:46.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:46.631+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:46.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:46.632+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:46.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:46.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:46.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:46.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:46.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:46.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:46.776+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:47.337 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:47.078+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9749 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:47.669 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:47.335+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:47.670 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:47.335+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:47.670 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:47.335+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:47.670 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:47.336+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:47.758+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:47.758+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:47.758+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:47.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:47.760+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:47.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:47.667+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:47.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:47.667+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:47.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:47.667+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:47.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:47.668+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:48.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:48.368+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:48.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:48.368+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:48.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:48.368+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:48.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:48.369+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:48.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:48.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:48.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:48.717+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:48.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:48.629+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:48.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:48.629+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:48.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:48.629+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:48.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:48.630+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:49.590 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:49.336+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:49.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:49.336+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:49.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:49.336+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:49.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:49.337+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:49.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:49.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:49.679+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:49.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:49.680+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:49.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:49.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:49.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:49.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:49.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:49.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:49.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:49.589+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:50.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:50.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:50.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:50.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:50.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:50.349+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:50.554 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:50.350+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:50.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:50.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:50.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:50.707+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:50.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:50.551+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:50.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:50.551+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:50.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:50.551+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:50.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:50.552+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:51.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:51.358+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:51.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:51.358+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:51.671 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:51.358+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:51.672 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:51.359+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:51.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:51.600+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:51.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:51.600+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:51.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:51.600+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:51.672 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:51.601+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:51.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:51.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:51.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:51.746+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:52.382 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:52.079+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9837 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:52.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:52.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:52.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:52.584+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:52.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:52.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:52.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:52.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:52.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:52.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:52.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:52.380+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:52.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:52.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:52.776+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:52.777+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:53.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:53.427+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:53.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:53.427+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:53.570 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:53.427+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:53.571 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:53.428+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:53.589+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:53.589+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:53.589+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:53.589+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:53.768+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:53.768+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:53.768+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:53.769+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:54.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:54.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:54.379+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:54.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:54.379+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:54.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:54.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:54.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:54.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:54.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:54.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:54.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:54.551+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:54.780+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:54.780+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:54.780+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:54.781+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:55.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:55.333+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:55.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:55.333+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:55.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:55.333+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:55.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:55.334+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:55.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:55.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:55.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:55.785+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:55.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:55.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:55.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:55.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:55.586+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:56.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:56.593+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:56.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:56.593+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:56.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:56.593+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:56.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:56.594+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:56.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:56.375+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:56.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:56.375+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:56.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:56.375+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:56.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:56.376+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:57.083 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:56.779+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:57.083 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:56.779+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:57.083 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:56.779+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:57.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:56.780+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:57.418 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:00:57.081+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 9935 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:00:57.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:57.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:57.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:57.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:57.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:57.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:57.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:57.416+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:57.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:57.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:57.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:57.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:57.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:57.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:57.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:57.639+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:57.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:57.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:57.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:57.795+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:58.799 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:58.669+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:58.799 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:58.669+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:58.799 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:58.669+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:00:58.800 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:58.670+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:58.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:58.458+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:58.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:58.458+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:58.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:58.458+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:58.800 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:58.458+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:00:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:58.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:58.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:58.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:00:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:58.798+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:00:59.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:59.433+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:59.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:59.433+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:59.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:59.433+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:00:59.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:00:59.434+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:59.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:59.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:00.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:59.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:00.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:00:59.701+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:00.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:59.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:00.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:59.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:00.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:59.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:00.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:00:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:00:59.785+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:00.735 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:00.479+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:00.735 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:00.479+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:00.735 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:00.479+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:00.736 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:00.479+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:01.090 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:00.817+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:01.091 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:00.817+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:01.091 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:00.817+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:01.091 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:00.818+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:01.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:00.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:01.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:00.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:01.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:00.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:01.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:00.733+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:01.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:01.698+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:01.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:01.698+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:01.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:01.698+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:01.797 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:01.698+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:01.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:01.497+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:01.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:01.497+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:01.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:01.497+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:01.797 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:01.499+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:02.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:01.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:02.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:01.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:02.084 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:01.794+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:02.085 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:01.795+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:02.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:02.082+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10029 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:02.766 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:02.471+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:02.767 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:02.471+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:02.767 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:02.472+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:02.767 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:02.473+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:02.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:02.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:02.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:02.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:02.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:02.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:02.767 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:02.674+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:02.764+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:02.764+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:03.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:02.764+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:03.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:02.765+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:03.508+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:03.508+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:03.508+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:03.509+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:03.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:03.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:03.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:03.694+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:03.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:03.735+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:03.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:03.735+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:03.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:03.735+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:03.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:03.736+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:04.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:04.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:04.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:04.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:04.717+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:04.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:04.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:04.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:04.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:04.727+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:04.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:04.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:04.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:04.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:04.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:04.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:04.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:04.527+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:05.767+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:05.767+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:05.767+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:05.768+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:05.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:05.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:05.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:05.747+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:05.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:05.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:05.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:05.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:05.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:05.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:05.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:05.552+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:06.737+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:06.737+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:06.737+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:06.738+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:06.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:06.524+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:06.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:06.524+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:06.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:06.524+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:06.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:06.525+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:06.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:06.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:06.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:06.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:06.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:06.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:06.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:06.743+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:07.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:07.083+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10127 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:07.713+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:07.713+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:07.713+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:07.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:07.714+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:07.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:07.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:07.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:07.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:07.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:07.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:07.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:07.743+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:07.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:07.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:07.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:07.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:07.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:07.526+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:07.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:07.527+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:08.751+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:08.752+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:08.752+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:08.753+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:08.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:08.784+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:08.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:08.784+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:08.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:08.784+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:08.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:08.785+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:08.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:08.553+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:08.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:08.553+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:08.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:08.553+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:08.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:08.553+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:09.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:09.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:09.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:09.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:09.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:09.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:09.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:09.601+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:09.827+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:09.827+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:09.827+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:09.827+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:09.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:09.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:09.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:09.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:09.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:09.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:09.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:09.722+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:10.838 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:10.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:10.840 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:10.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:10.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:10.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:10.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:10.683+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:10.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:10.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:10.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:10.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:10.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:10.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:10.841 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:10.571+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:10.835+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:10.835+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:10.835+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:10.836+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:11.565+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:11.565+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:11.565+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:11.566+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:11.806+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:11.806+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:11.806+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:11.806+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:11.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:11.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:11.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:11.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:11.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:11.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:11.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:11.642+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:12.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:12.084+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10222 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:12.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:12.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:12.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:12.667+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:12.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:12.605+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:12.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:12.605+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:12.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:12.605+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:12.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:12.606+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:12.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:12.836+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:12.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:12.836+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:12.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:12.836+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:12.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:12.837+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:13.827 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:13.560+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:13.827 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:13.560+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:13.827 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:13.560+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:13.827 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:13.561+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:13.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:13.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:13.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:13.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:13.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:13.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:13.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:13.641+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:14.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:13.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:14.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:13.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:14.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:13.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:14.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:13.825+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:14.863 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:14.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:14.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:14.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:14.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:14.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:14.864 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:14.591+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:14.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:14.595+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:14.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:14.595+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:14.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:14.595+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:14.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:14.597+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:14.868+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:14.868+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:14.868+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:14.869+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:15.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:15.587+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:15.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:15.587+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:15.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:15.587+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:15.874 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:15.588+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:15.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:15.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:15.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:15.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:15.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:15.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:15.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:15.570+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:15.871+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:15.871+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:15.871+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:15.871+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:16.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:16.542+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:16.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:16.542+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:16.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:16.542+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:16.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:16.543+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:16.917 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:16.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:16.917 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:16.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:16.917 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:16.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:16.917 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:16.590+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:17.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:17.085+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10321 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:17.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:16.913+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:17.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:16.913+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:17.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:16.913+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:17.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:16.914+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:17.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:17.566+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:17.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:17.567+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:17.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:17.567+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:17.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:17.567+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:17.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:17.561+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:17.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:17.561+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:17.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:17.561+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:17.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:17.562+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:18.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:17.945+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:18.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:17.945+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:18.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:17.945+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:18.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:17.946+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:18.944 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:18.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:18.944 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:18.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:18.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:18.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:18.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:18.549+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:18.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:18.526+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:18.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:18.526+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:18.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:18.526+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:18.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:18.527+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:19.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:18.942+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:19.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:18.942+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:19.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:18.942+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:19.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:18.943+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:19.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:19.555+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:19.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:19.555+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:19.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:19.555+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:19.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:19.556+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:19.939 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:19.546+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:19.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:19.546+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:19.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:19.546+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:19.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:19.547+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:20.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:19.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:20.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:19.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:20.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:19.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:20.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:19.937+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:20.530+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:20.530+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:20.530+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:20.531+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:20.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:20.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:20.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:20.569+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:21.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:20.951+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:21.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:20.951+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:21.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:20.951+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:21.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:20.951+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:21.552+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:21.552+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:21.552+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:21.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:21.553+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:21.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:21.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:21.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:21.541+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:22.214 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:22.087+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10415 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:22.215 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:21.960+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:22.215 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:21.960+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:22.215 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:21.960+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:22.215 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:21.961+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:22.521+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:22.521+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:22.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:22.521+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:22.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:22.522+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:22.502+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:22.502+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:22.502+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:22.503+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:22.971+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:22.971+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:22.971+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:22.972+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:23.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:23.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:23.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:23.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:23.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:23.549+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:23.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:23.549+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:23.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:23.549+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:23.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:23.549+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:23.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:23.550+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:24.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:23.992+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:24.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:23.992+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:24.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:23.992+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:24.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:23.993+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:24.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:24.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:24.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:24.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:24.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:24.577+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:24.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:24.558+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:24.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:24.558+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:24.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:24.558+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:24.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:24.559+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:24.995+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:25.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:24.995+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:25.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:24.995+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:25.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:24.995+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:25.570+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:25.570+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:25.570+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:25.571+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:25.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:25.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:25.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:25.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:25.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:25.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:25.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:25.569+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:25.960+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:25.960+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:25.960+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:26.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:25.961+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:26.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:26.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:26.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:26.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:26.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:26.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:26.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:26.570+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:26.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:26.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:26.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:26.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:26.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:26.618+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:26.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:26.619+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:27.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:27.088+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10513 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:27.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:26.932+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:27.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:26.932+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:27.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:26.932+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:27.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:26.933+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:27.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:27.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:27.640+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:27.641+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:27.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:27.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:27.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:27.576+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:27.953+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:27.953+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:27.953+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:27.953+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:28.549+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:28.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:28.549+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:28.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:28.549+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:28.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:28.549+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:28.613+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:28.613+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:28.613+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:28.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:28.614+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:29.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:28.958+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:29.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:28.958+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:29.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:28.958+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:29.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:28.959+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:29.925 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:29.636+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:29.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:29.636+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:29.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:29.636+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:29.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:29.637+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:29.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:29.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:29.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:29.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:29.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:29.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:29.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:29.549+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:30.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:29.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:30.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:29.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:30.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:29.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:30.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:29.923+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:30.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:30.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:30.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:30.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:30.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:30.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:30.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:30.660+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:30.509+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:30.509+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:30.509+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:30.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:30.509+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:31.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:30.947+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:31.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:30.947+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:31.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:30.947+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:31.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:30.947+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:31.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:31.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:31.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:31.661+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:31.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:31.532+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:31.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:31.532+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:31.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:31.532+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:31.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:31.533+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:32.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:31.948+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:32.216 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:31.948+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:32.217 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:31.948+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:32.217 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:31.949+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:32.217 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:32.089+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10607 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:32.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:32.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:32.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:32.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:32.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:32.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:32.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:32.554+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:32.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:32.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:32.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:32.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:32.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:32.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:32.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:32.617+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:33.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:32.931+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:33.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:32.931+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:33.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:32.931+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:33.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:32.931+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:33.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:33.510+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:33.573 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:33.510+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:33.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:33.510+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:33.574 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:33.511+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:33.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:33.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:33.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:33.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:33.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:33.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:33.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:33.573+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:34.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:33.939+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:34.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:33.939+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:34.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:33.939+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:34.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:33.940+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:34.904 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:34.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:34.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:34.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:34.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:34.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:34.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:34.475+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:34.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:34.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:34.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:34.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:34.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:34.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:34.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:34.535+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:34.900+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:34.900+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:34.900+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:34.901+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:35.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:35.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:35.548+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:35.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:35.549+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:35.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:35.445+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:35.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:35.445+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:35.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:35.445+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:35.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:35.446+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:35.870+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:35.870+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:35.870+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:35.871+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:36.471+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:36.471+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:36.471+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:36.472+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:36.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:36.508+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:36.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:36.508+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:36.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:36.508+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:36.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:36.508+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:37.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:37.090+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10706 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:37.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:36.874+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:37.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:36.874+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:37.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:36.874+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:37.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:36.875+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:37.496 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:37.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:37.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:37.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:37.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:37.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:37.497 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:37.445+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:37.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:37.494+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:37.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:37.494+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:37.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:37.494+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:37.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:37.495+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:37.912+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:37.912+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:37.912+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:38.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:37.913+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:38.505+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:38.505+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:38.505+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:38.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:38.506+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:38.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:38.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:38.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:38.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:38.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:38.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:38.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:38.406+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:38.886+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:38.886+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:38.886+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:38.887+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:39.540 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:39.443+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:39.541 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:39.443+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:39.541 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:39.443+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:39.541 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:39.444+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:39.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:39.538+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:39.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:39.538+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:39.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:39.538+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:39.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:39.539+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:40.143 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:39.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:40.143 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:39.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:40.143 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:39.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:40.143 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:39.860+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:40.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:40.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:40.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:40.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:40.578+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:40.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:40.437+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:40.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:40.437+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:40.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:40.437+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:40.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:40.438+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:40.855+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:40.855+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:40.855+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:40.856+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:41.675 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:41.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:41.675 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:41.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:41.675 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:41.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:41.676 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:41.571+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:41.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:41.398+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:41.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:41.398+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:41.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:41.398+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:41.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:41.399+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:41.879+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:41.879+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:41.879+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:41.880+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:42.415 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:42.092+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10801 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:42.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:42.413+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:42.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:42.413+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:42.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:42.413+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:42.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:42.414+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:42.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:42.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:42.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:42.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:42.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:42.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:42.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:42.580+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:43.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:42.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:43.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:42.908+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:43.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:42.908+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:43.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:42.908+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:43.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:43.430+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:43.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:43.430+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:43.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:43.430+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:43.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:43.432+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:43.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:43.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:43.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:43.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:43.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:43.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:43.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:43.626+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:43.949+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:43.949+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:43.949+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:43.950+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:44.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:44.453+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:44.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:44.453+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:44.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:44.453+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:44.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:44.454+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:44.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:44.673+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:44.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:44.673+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:44.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:44.673+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:44.929 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:44.674+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:44.925+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:44.925+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:44.925+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:44.926+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:45.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:45.429+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:45.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:45.429+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:45.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:45.429+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:45.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:45.430+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:45.722+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:45.722+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:45.722+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:45.722+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:46.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:45.949+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:46.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:45.949+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:46.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:45.949+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:46.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:45.950+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:46.452+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:46.452+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:46.452+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:46.453+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:46.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:46.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:46.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:46.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:46.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:46.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:46.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:46.696+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:47.217 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:46.963+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:47.217 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:46.963+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:47.218 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:46.963+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:47.218 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:46.964+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:47.218 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:47.093+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10898 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:47.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:47.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:47.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:47.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:47.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:47.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:47.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:47.426+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:47.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:47.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:47.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:47.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:47.674+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:48.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:47.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:48.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:47.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:48.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:47.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:48.407 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:47.988+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:48.672 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:48.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:48.672 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:48.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:48.673 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:48.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:48.673 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:48.406+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:48.670+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:48.670+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:48.670+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:48.671+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:49.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:48.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:49.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:48.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:49.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:48.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:49.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:48.988+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:49.691 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:49.442+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:49.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:49.442+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:49.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:49.442+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:49.692 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:49.443+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:49.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:49.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:49.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:49.690+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:50.428 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:49.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:50.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:49.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:50.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:49.988+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:50.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:49.988+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:50.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:50.426+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:50.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:50.426+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:50.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:50.426+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:50.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:50.427+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:51.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:50.724+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:51.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:50.724+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:51.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:50.724+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:51.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:50.725+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:51.391 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:51.014+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:51.391 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:51.014+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:51.391 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:51.014+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:51.391 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:51.014+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:51.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:51.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:51.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:51.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:51.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:51.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:51.676 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:51.389+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:51.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:51.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:51.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:51.695+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:52.344 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:52.093+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 10991 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:52.345 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:52.028+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:52.345 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:52.028+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:52.345 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:52.028+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:52.345 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:52.028+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:52.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:52.341+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:52.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:52.341+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:52.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:52.341+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:52.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:52.342+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:52.694+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:52.694+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:52.694+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:52.695+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:53.306 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:53.011+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:53.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:53.011+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:53.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:53.011+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:53.307 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:53.011+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:53.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:53.304+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:53.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:53.304+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:53.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:53.304+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:53.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:53.304+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:53.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:53.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:53.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:53.652+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:54.353 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:54.053+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:54.353 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:54.053+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:54.353 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:54.053+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:54.353 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:54.053+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:54.652 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:54.350+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:54.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:54.350+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:54.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:54.350+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:54.653 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:54.350+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:54.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:54.650+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:54.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:54.650+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:54.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:54.650+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:54.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:54.650+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:55.308 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:55.024+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:55.308 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:55.024+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:55.308 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:55.024+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:55.309 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:55.025+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:55.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:55.305+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:55.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:55.305+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:55.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:55.305+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:55.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:55.306+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:55.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:55.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:55.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:55.640+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:56.264 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:55.979+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:56.264 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:55.979+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:56.264 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:55.979+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:56.264 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:55.980+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:56.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:56.262+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:56.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:56.262+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:56.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:56.262+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:56.629 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:56.263+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:56.626+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:56.626+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:56.626+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:56.627+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:57.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:01:57.096+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11089 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:01:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:56.995+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:56.995+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:56.995+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:56.996+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:57.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:57.240+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:57.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:57.240+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:57.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:57.240+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:57.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:57.241+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:57.871 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:57.584+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:57.871 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:57.584+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:57.871 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:57.584+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:57.871 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:57.585+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:58.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:57.949+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:58.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:57.949+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:58.266 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:57.949+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:58.267 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:57.950+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:58.616 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:58.264+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:58.616 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:58.264+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:58.616 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:58.264+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:58.616 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:58.265+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:58.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:58.613+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:58.937 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:58.613+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:58.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:58.613+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:58.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:58.614+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:01:59.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:58.935+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:59.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:58.935+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:59.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:58.935+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:01:59.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:58.936+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:59.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:59.239+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:59.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:59.239+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:59.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:59.239+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:01:59.658 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:01:59.240+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:01:59.914 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:59.655+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:59.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:59.655+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:59.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:59.655+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:01:59.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:01:59.656+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:59.912+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:59.912+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:59.912+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:00.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:01:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:01:59.913+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:00.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:00.217+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:00.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:00.217+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:00.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:00.217+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:00.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:00.218+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:00.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:00.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:00.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:00.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:00.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:00.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:00.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:00.655+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:00.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:00.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:00.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:00.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:00.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:00.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:00.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:00.865+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:01.174+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:01.174+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:01.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:01.174+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:01.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:01.175+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:01.626+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:01.626+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:01.626+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:01.627+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:01.861+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:01.861+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:01.861+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:01.862+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:02.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:02.097+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11183 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:02.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:02.190+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:02.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:02.190+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:02.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:02.190+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:02.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:02.190+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:02.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:02.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:02.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:02.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:02.882 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:02.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:02.882 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:02.600+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:03.183 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:02.879+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:03.183 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:02.879+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:03.183 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:02.879+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:03.183 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:02.879+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:03.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:03.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:03.180+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:03.181+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:03.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:03.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:03.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:03.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:03.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:03.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:03.881 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:03.589+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:04.140 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:03.878+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:04.140 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:03.878+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:04.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:03.878+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:04.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:03.879+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:04.138+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:04.138+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:04.138+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:04.139+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:04.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:04.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:04.923 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:04.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:04.923 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:04.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:04.923 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:04.541+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:05.163+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:05.163+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:05.163+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:05.163+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:05.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:04.920+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:05.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:04.920+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:05.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:04.920+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:05.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:04.920+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:05.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:05.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:05.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:05.541+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:06.117+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:06.117+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:06.117+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:06.117+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:06.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:05.952+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:06.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:05.952+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:06.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:05.952+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:06.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:05.952+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:06.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:06.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:06.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:06.534+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:06.535+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:07.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:07.098+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11281 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:07.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:06.947+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:07.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:06.947+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:07.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:06.947+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:07.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:06.948+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:07.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:07.073+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:07.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:07.073+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:07.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:07.073+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:07.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:07.074+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:07.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:07.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:07.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:07.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:07.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:07.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:07.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:07.552+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:08.071+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:08.071+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:08.071+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:08.071+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:08.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:07.928+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:08.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:07.928+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:08.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:07.928+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:08.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:07.929+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:08.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:08.561+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:08.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:08.561+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:08.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:08.561+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:08.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:08.562+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:09.057+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:09.057+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:09.057+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:09.058+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:09.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:08.938+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:08.938+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:08.938+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:08.939+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:09.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:09.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:09.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:09.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:09.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:09.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:09.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:09.578+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:10.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:10.020+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:10.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:10.020+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:10.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:10.020+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:10.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:10.021+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:10.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:09.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:10.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:09.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:10.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:09.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:10.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:09.916+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:10.899 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:10.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:10.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:10.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:10.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:10.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:10.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:10.597+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:10.897+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:10.897+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:10.897+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:10.898+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:10.994+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:10.994+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:10.994+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:10.994+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:11.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:11.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:11.895 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:11.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:11.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:11.572+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:11.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:11.574+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:12.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:12.100+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11375 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:12.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:11.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:12.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:11.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:12.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:11.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:12.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:11.894+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:12.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:12.006+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:12.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:12.006+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:12.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:12.006+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:12.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:12.007+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:12.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:12.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:12.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:12.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:12.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:12.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:12.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:12.575+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:12.974+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:12.974+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:12.974+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:12.974+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:13.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:12.873+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:13.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:12.873+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:13.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:12.873+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:13.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:12.874+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:13.854 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:13.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:13.854 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:13.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:13.854 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:13.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:13.854 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:13.622+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:13.936+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:13.936+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:13.936+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:14.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:13.937+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:14.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:13.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:14.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:13.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:14.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:13.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:14.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:13.852+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:14.642+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:14.642+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:14.642+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:14.644+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:14.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:14.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:14.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:14.803+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:14.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:14.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:14.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:14.952+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:15.792+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:15.792+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:15.792+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:15.792+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:15.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:15.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:15.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:15.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:15.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:15.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:15.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:15.603+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:16.302 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:15.978+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:16.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:15.978+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:16.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:15.978+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:16.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:15.980+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:16.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:16.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:16.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:16.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:16.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:16.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:16.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:16.646+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:16.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:16.759+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:16.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:16.759+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:16.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:16.759+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:16.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:16.759+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:17.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:17.101+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11473 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:17.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:17.014+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:17.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:17.014+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:17.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:17.014+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:17.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:17.015+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:17.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:17.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:17.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:17.750+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:17.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:17.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:17.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:17.600+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:17.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:17.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:17.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:17.967+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:18.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:18.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:18.921 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:18.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:18.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:18.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:18.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:18.645+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:18.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:18.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:18.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:18.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:18.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:18.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:18.922 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:18.748+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:18.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:18.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:18.918+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:18.920+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:19.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:19.597+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:19.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:19.597+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:19.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:19.597+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:19.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:19.598+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:19.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:19.775+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:19.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:19.775+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:19.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:19.775+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:19.893 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:19.776+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:20.146 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:19.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:20.146 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:19.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:20.146 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:19.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:20.147 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:19.890+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:20.902 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:20.598+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:20.902 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:20.598+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:20.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:20.598+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:20.903 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:20.599+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:20.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:20.761+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:20.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:20.761+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:20.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:20.761+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:20.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:20.762+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:20.900+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:20.900+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:20.900+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:20.901+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:21.866 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:21.560+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:21.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:21.560+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:21.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:21.560+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:21.867 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:21.561+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:21.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:21.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:21.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:21.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:21.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:21.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:21.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:21.738+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:22.116 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:22.103+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11567 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:22.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:21.864+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:22.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:21.864+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:22.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:21.864+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:22.117 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:21.865+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-19T19:02:22.264 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-12-19T19:02:22.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:02:22.264 DEBUG:teuthology.orchestra.run.smithi059:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-12-19T19:02:22.306 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/accept: Cannot mkdir: Input/output error 2023-12-19T19:02:22.306 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/accept/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.306 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/accept/accept01.c: Cannot open: Input/output error 2023-12-19T19:02:22.306 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/abort: Cannot mkdir: Input/output error 2023-12-19T19:02:22.306 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/abort/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.306 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/abort/abort01.c: Cannot open: Input/output error 2023-12-19T19:02:22.307 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups: Cannot mkdir: Input/output error 2023-12-19T19:02:22.307 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/getgroups04.c: Cannot open: Input/output error 2023-12-19T19:02:22.307 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.307 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/getgroups02.c: Cannot open: Input/output error 2023-12-19T19:02:22.307 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/getgroups01.c: Cannot open: Input/output error 2023-12-19T19:02:22.307 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgroups/getgroups03.c: Cannot open: Input/output error 2023-12-19T19:02:22.307 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmodat: Cannot mkdir: Input/output error 2023-12-19T19:02:22.307 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmodat/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.307 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fchmodat/fchmodat01.c: Cannot open: Input/output error 2023-12-19T19:02:22.316 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_setup: Cannot mkdir: Input/output error 2023-12-19T19:02:22.316 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_setup/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.316 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_setup/io_setup01.c: Cannot open: Input/output error 2023-12-19T19:02:22.316 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask: Cannot mkdir: Input/output error 2023-12-19T19:02:22.316 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask/umask01.c: Cannot open: Input/output error 2023-12-19T19:02:22.316 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask/umask03.c: Cannot open: Input/output error 2023-12-19T19:02:22.317 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.317 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/umask/umask02.c: Cannot open: Input/output error 2023-12-19T19:02:22.317 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknodat: Cannot mkdir: Input/output error 2023-12-19T19:02:22.317 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknodat/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.317 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mknodat/mknodat01.c: Cannot open: Input/output error 2023-12-19T19:02:22.317 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsockname: Cannot mkdir: Input/output error 2023-12-19T19:02:22.317 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsockname/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.317 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getsockname/getsockname01.c: Cannot open: Input/output error 2023-12-19T19:02:22.318 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/splice: Cannot mkdir: Input/output error 2023-12-19T19:02:22.318 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/splice/splice01.c: Cannot open: Input/output error 2023-12-19T19:02:22.318 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/splice/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.318 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/splice/splice02.c: Cannot open: Input/output error 2023-12-19T19:02:22.318 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid: Cannot mkdir: Input/output error 2023-12-19T19:02:22.319 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/compat_16.h: Cannot open: Input/output error 2023-12-19T19:02:22.319 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/setgid01.c: Cannot open: Input/output error 2023-12-19T19:02:22.319 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.319 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/setgid02.c: Cannot open: Input/output error 2023-12-19T19:02:22.319 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setgid/setgid03.c: Cannot open: Input/output error 2023-12-19T19:02:22.319 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup3: Cannot mkdir: Input/output error 2023-12-19T19:02:22.319 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup3/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.320 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/dup3/dup3_01.c: Cannot open: Input/output error 2023-12-19T19:02:22.320 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid: Cannot mkdir: Input/output error 2023-12-19T19:02:22.320 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.320 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/setregid02.c: Cannot open: Input/output error 2023-12-19T19:02:22.320 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/setregid03.c: Cannot open: Input/output error 2023-12-19T19:02:22.320 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/setregid01.c: Cannot open: Input/output error 2023-12-19T19:02:22.321 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setregid/setregid04.c: Cannot open: Input/output error 2023-12-19T19:02:22.321 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/prctl: Cannot mkdir: Input/output error 2023-12-19T19:02:22.321 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/prctl/prctl02.c: Cannot open: Input/output error 2023-12-19T19:02:22.321 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/prctl/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.321 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/prctl/prctl01.c: Cannot open: Input/output error 2023-12-19T19:02:22.321 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigwait: Cannot mkdir: Input/output error 2023-12-19T19:02:22.321 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigwait/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.322 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/exit_group: Cannot mkdir: Input/output error 2023-12-19T19:02:22.322 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/exit_group/exit_group01.c: Cannot open: Input/output error 2023-12-19T19:02:22.322 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/exit_group/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.322 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fdatasync: Cannot mkdir: Input/output error 2023-12-19T19:02:22.322 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fdatasync/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.322 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fdatasync/fdatasync02.c: Cannot open: Input/output error 2023-12-19T19:02:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fdatasync/fdatasync01.c: Cannot open: Input/output error 2023-12-19T19:02:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/personality: Cannot mkdir: Input/output error 2023-12-19T19:02:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/personality/personality02.c: Cannot open: Input/output error 2023-12-19T19:02:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/personality/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/personality/personality01.c: Cannot open: Input/output error 2023-12-19T19:02:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pselect: Cannot mkdir: Input/output error 2023-12-19T19:02:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pselect/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pselect/pselect01.c: Cannot open: Input/output error 2023-12-19T19:02:22.323 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount: Cannot mkdir: Input/output error 2023-12-19T19:02:22.324 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/mount01.c: Cannot open: Input/output error 2023-12-19T19:02:22.324 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/setuid_test.c: Cannot open: Input/output error 2023-12-19T19:02:22.324 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.324 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/mount03.c: Cannot open: Input/output error 2023-12-19T19:02:22.324 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/mount04.c: Cannot open: Input/output error 2023-12-19T19:02:22.324 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/test_mount: Cannot open: Input/output error 2023-12-19T19:02:22.324 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mount/mount02.c: Cannot open: Input/output error 2023-12-19T19:02:22.324 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_get_priority_max: Cannot mkdir: Input/output error 2023-12-19T19:02:22.325 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_get_priority_max/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.325 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_get_priority_max/sched_get_priority_max02.c: Cannot open: Input/output error 2023-12-19T19:02:22.325 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_get_priority_max/sched_get_priority_max01.c: Cannot open: Input/output error 2023-12-19T19:02:22.325 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rt_sigsuspend: Cannot mkdir: Input/output error 2023-12-19T19:02:22.325 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rt_sigsuspend/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.325 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rt_sigsuspend/rt_sigsuspend01.c: Cannot open: Input/output error 2023-12-19T19:02:22.325 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid: Cannot mkdir: Input/output error 2023-12-19T19:02:22.325 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/setresuid04.c: Cannot open: Input/output error 2023-12-19T19:02:22.326 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.326 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/setresuid02.c: Cannot open: Input/output error 2023-12-19T19:02:22.326 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/setresuid01.c: Cannot open: Input/output error 2023-12-19T19:02:22.326 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresuid/setresuid03.c: Cannot open: Input/output error 2023-12-19T19:02:22.326 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam: Cannot mkdir: Input/output error 2023-12-19T19:02:22.326 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam05.c: Cannot open: Input/output error 2023-12-19T19:02:22.326 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.326 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam04.c: Cannot open: Input/output error 2023-12-19T19:02:22.327 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam03.c: Cannot open: Input/output error 2023-12-19T19:02:22.327 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam02.c: Cannot open: Input/output error 2023-12-19T19:02:22.327 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_setparam/sched_setparam01.c: Cannot open: Input/output error 2023-12-19T19:02:22.327 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/recvmsg: Cannot mkdir: Input/output error 2023-12-19T19:02:22.327 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/recvmsg/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.327 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/recvmsg/recvmsg01.c: Cannot open: Input/output error 2023-12-19T19:02:22.327 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ulimit: Cannot mkdir: Input/output error 2023-12-19T19:02:22.327 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ulimit/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ulimit/README: Cannot open: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ulimit/ulimit01.c: Cannot open: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid: Cannot mkdir: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid/setfsgid02.c: Cannot open: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid/setfsgid03.c: Cannot open: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setfsgid/setfsgid01.c: Cannot open: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink: Cannot mkdir: Input/output error 2023-12-19T19:02:22.328 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/unlink05.c: Cannot open: Input/output error 2023-12-19T19:02:22.329 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.329 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/unlink07.c: Cannot open: Input/output error 2023-12-19T19:02:22.329 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/unlink08.c: Cannot open: Input/output error 2023-12-19T19:02:22.329 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/unlink/unlink06.c: Cannot open: Input/output error 2023-12-19T19:02:22.329 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgrp: Cannot mkdir: Input/output error 2023-12-19T19:02:22.329 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgrp/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.329 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgrp/setpgrp02.c: Cannot open: Input/output error 2023-12-19T19:02:22.329 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setpgrp/setpgrp01.c: Cannot open: Input/output error 2023-12-19T19:02:22.329 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid: Cannot mkdir: Input/output error 2023-12-19T19:02:22.330 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid04.c: Cannot open: Input/output error 2023-12-19T19:02:22.330 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid10.c: Cannot open: Input/output error 2023-12-19T19:02:22.330 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid09.c: Cannot open: Input/output error 2023-12-19T19:02:22.330 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid07.c: Cannot open: Input/output error 2023-12-19T19:02:22.330 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid02.c: Cannot open: Input/output error 2023-12-19T19:02:22.330 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid01.c: Cannot open: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid13.c: Cannot open: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid05.c: Cannot open: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid08.c: Cannot open: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid11.c: Cannot open: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid03.c: Cannot open: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid12.c: Cannot open: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/waitpid/waitpid06.c: Cannot open: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_destroy: Cannot mkdir: Input/output error 2023-12-19T19:02:22.331 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_destroy/io_destroy01.c: Cannot open: Input/output error 2023-12-19T19:02:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_destroy/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat: Cannot mkdir: Input/output error 2023-12-19T19:02:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat/ustat02.c: Cannot open: Input/output error 2023-12-19T19:02:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat/ustat01.c: Cannot open: Input/output error 2023-12-19T19:02:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigsuspend: Cannot mkdir: Input/output error 2023-12-19T19:02:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigsuspend/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.332 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigsuspend/sigsuspend01.c: Cannot open: Input/output error 2023-12-19T19:02:22.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_unlink: Cannot mkdir: Input/output error 2023-12-19T19:02:22.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_unlink/mq_unlink01.c: Cannot open: Input/output error 2023-12-19T19:02:22.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_unlink/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcpu: Cannot mkdir: Input/output error 2023-12-19T19:02:22.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcpu/getcpu01.c: Cannot open: Input/output error 2023-12-19T19:02:22.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcpu/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.333 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2: Cannot mkdir: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2/pipe2_01.c: Cannot open: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2/pipe2_02.c: Cannot open: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir: Cannot mkdir: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir09.c: Cannot open: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir08.c: Cannot open: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir01.c: Cannot open: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir04.c: Cannot open: Input/output error 2023-12-19T19:02:22.334 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir05.c: Cannot open: Input/output error 2023-12-19T19:02:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir02.c: Cannot open: Input/output error 2023-12-19T19:02:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir03.c: Cannot open: Input/output error 2023-12-19T19:02:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname: Cannot mkdir: Input/output error 2023-12-19T19:02:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/uname01.c: Cannot open: Input/output error 2023-12-19T19:02:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/uname03.c: Cannot open: Input/output error 2023-12-19T19:02:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/uname02.c: Cannot open: Input/output error 2023-12-19T19:02:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid: Cannot mkdir: Input/output error 2023-12-19T19:02:22.335 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/getegid02.c: Cannot open: Input/output error 2023-12-19T19:02:22.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/compat_16.h: Cannot open: Input/output error 2023-12-19T19:02:22.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/getegid01.c: Cannot open: Input/output error 2023-12-19T19:02:22.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read: Cannot mkdir: Input/output error 2023-12-19T19:02:22.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read03.c: Cannot open: Input/output error 2023-12-19T19:02:22.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read02.c: Cannot open: Input/output error 2023-12-19T19:02:22.336 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read04.c: Cannot open: Input/output error 2023-12-19T19:02:22.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read01.c: Cannot open: Input/output error 2023-12-19T19:02:22.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock: Cannot mkdir: Input/output error 2023-12-19T19:02:22.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock/mlock01.c: Cannot open: Input/output error 2023-12-19T19:02:22.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock/mlock02.c: Cannot open: Input/output error 2023-12-19T19:02:22.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/poll: Cannot mkdir: Input/output error 2023-12-19T19:02:22.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/poll/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.337 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/poll/poll01.c: Cannot open: Input/output error 2023-12-19T19:02:22.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_cancel: Cannot mkdir: Input/output error 2023-12-19T19:02:22.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_cancel/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_cancel/io_cancel01.c: Cannot open: Input/output error 2023-12-19T19:02:22.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execlp: Cannot mkdir: Input/output error 2023-12-19T19:02:22.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execlp/execlp01.c: Cannot open: Input/output error 2023-12-19T19:02:22.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execlp/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat: Cannot mkdir: Input/output error 2023-12-19T19:02:22.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat06.c: Cannot open: Input/output error 2023-12-19T19:02:22.338 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat02.c: Cannot open: Input/output error 2023-12-19T19:02:22.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat03.c: Cannot open: Input/output error 2023-12-19T19:02:22.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat05.c: Cannot open: Input/output error 2023-12-19T19:02:22.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat01.c: Cannot open: Input/output error 2023-12-19T19:02:22.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs: Cannot mkdir: Input/output error 2023-12-19T19:02:22.339 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.340 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs/fstatfs01.c: Cannot open: Input/output error 2023-12-19T19:02:22.340 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs/fstatfs02.c: Cannot open: Input/output error 2023-12-19T19:02:22.340 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clock_nanosleep: Cannot mkdir: Input/output error 2023-12-19T19:02:22.340 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clock_nanosleep/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.341 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clock_nanosleep/clock_nanosleep01.c: Cannot open: Input/output error 2023-12-19T19:02:22.341 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink: Cannot mkdir: Input/output error 2023-12-19T19:02:22.341 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink02.c: Cannot open: Input/output error 2023-12-19T19:02:22.341 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink01.c: Cannot open: Input/output error 2023-12-19T19:02:22.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink05.c: Cannot open: Input/output error 2023-12-19T19:02:22.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink03.c: Cannot open: Input/output error 2023-12-19T19:02:22.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink04.c: Cannot open: Input/output error 2023-12-19T19:02:22.342 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect: Cannot mkdir: Input/output error 2023-12-19T19:02:22.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/mprotect01.c: Cannot open: Input/output error 2023-12-19T19:02:22.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/mprotect02.c: Cannot open: Input/output error 2023-12-19T19:02:22.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.343 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/mprotect03.c: Cannot open: Input/output error 2023-12-19T19:02:22.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir: Cannot mkdir: Input/output error 2023-12-19T19:02:22.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir02.c: Cannot open: Input/output error 2023-12-19T19:02:22.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir04.c: Cannot open: Input/output error 2023-12-19T19:02:22.344 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.345 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi059.front.sepia.ceph.com 2023-12-19T19:02:22.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:02:22.345 DEBUG:teuthology.orchestra.run.smithi059:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T19:02:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-19T19:02:22.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir03.c: Cannot open: Input/output error 2023-12-19T19:02:22.351 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir05.c: Cannot open: Input/output error 2023-12-19T19:02:22.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir01.c: Cannot open: Input/output error 2023-12-19T19:02:22.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction: Cannot mkdir: Input/output error 2023-12-19T19:02:22.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction/sigaction01.c: Cannot open: Input/output error 2023-12-19T19:02:22.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction/sigaction02.c: Cannot open: Input/output error 2023-12-19T19:02:22.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw: Cannot mkdir: Input/output error 2023-12-19T19:02:22.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/lib.c: Cannot open: Input/output error 2023-12-19T19:02:22.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw.h: Cannot open: Input/output error 2023-12-19T19:02:22.352 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/lib64.c: Cannot open: Input/output error 2023-12-19T19:02:22.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test.c: Cannot open: Input/output error 2023-12-19T19:02:22.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test_func.c: Cannot open: Input/output error 2023-12-19T19:02:22.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test_func64.c: Cannot open: Input/output error 2023-12-19T19:02:22.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw64.c: Cannot open: Input/output error 2023-12-19T19:02:22.353 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test64.c: Cannot open: Input/output error 2023-12-19T19:02:22.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw64.h: Cannot open: Input/output error 2023-12-19T19:02:22.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/tools.c: Cannot open: Input/output error 2023-12-19T19:02:22.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw.c: Cannot open: Input/output error 2023-12-19T19:02:22.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/tools64.c: Cannot open: Input/output error 2023-12-19T19:02:22.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedsend: Cannot mkdir: Input/output error 2023-12-19T19:02:22.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedsend/mq_timedsend01.c: Cannot open: Input/output error 2023-12-19T19:02:22.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedsend/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.354 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statvfs: Cannot mkdir: Input/output error 2023-12-19T19:02:22.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statvfs/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statvfs/statvfs01.c: Cannot open: Input/output error 2023-12-19T19:02:22.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot: Cannot mkdir: Input/output error 2023-12-19T19:02:22.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot/reboot01.c: Cannot open: Input/output error 2023-12-19T19:02:22.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot/reboot02.c: Cannot open: Input/output error 2023-12-19T19:02:22.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/confstr: Cannot mkdir: Input/output error 2023-12-19T19:02:22.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/confstr/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.355 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/confstr/confstr01.c: Cannot open: Input/output error 2023-12-19T19:02:22.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo: Cannot mkdir: Input/output error 2023-12-19T19:02:22.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo/sysinfo02.c: Cannot open: Input/output error 2023-12-19T19:02:22.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo/sysinfo01.c: Cannot open: Input/output error 2023-12-19T19:02:22.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execv: Cannot mkdir: Input/output error 2023-12-19T19:02:22.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execv/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execv/execv01.c: Cannot open: Input/output error 2023-12-19T19:02:22.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/cacheflush: Cannot mkdir: Input/output error 2023-12-19T19:02:22.356 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/cacheflush/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/cacheflush/cacheflush01.c: Cannot open: Input/output error 2023-12-19T19:02:22.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep: Cannot mkdir: Input/output error 2023-12-19T19:02:22.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep01.c: Cannot open: Input/output error 2023-12-19T19:02:22.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep04.c: Cannot open: Input/output error 2023-12-19T19:02:22.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep03.c: Cannot open: Input/output error 2023-12-19T19:02:22.357 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep02.c: Cannot open: Input/output error 2023-12-19T19:02:22.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_open: Cannot mkdir: Input/output error 2023-12-19T19:02:22.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_open/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_open/mq_open01.c: Cannot open: Input/output error 2023-12-19T19:02:22.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid: Cannot mkdir: Input/output error 2023-12-19T19:02:22.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/compat_16.h: Cannot open: Input/output error 2023-12-19T19:02:22.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/getgid03.c: Cannot open: Input/output error 2023-12-19T19:02:22.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.358 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/getgid01.c: Cannot open: Input/output error 2023-12-19T19:02:22.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup: Cannot mkdir: Input/output error 2023-12-19T19:02:22.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup/vhangup02.c: Cannot open: Input/output error 2023-12-19T19:02:22.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup/vhangup01.c: Cannot open: Input/output error 2023-12-19T19:02:22.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gethostname: Cannot mkdir: Input/output error 2023-12-19T19:02:22.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gethostname/gethostname01.c: Cannot open: Input/output error 2023-12-19T19:02:22.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gethostname/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct: Cannot mkdir: Input/output error 2023-12-19T19:02:22.359 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct/acct02.c: Cannot open: Input/output error 2023-12-19T19:02:22.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct/acct01.c: Cannot open: Input/output error 2023-12-19T19:02:22.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler: Cannot mkdir: Input/output error 2023-12-19T19:02:22.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler/sched_getscheduler02.c: Cannot open: Input/output error 2023-12-19T19:02:22.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler/sched_getscheduler01.c: Cannot open: Input/output error 2023-12-19T19:02:22.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone: Cannot mkdir: Input/output error 2023-12-19T19:02:22.360 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone02.c: Cannot open: Input/output error 2023-12-19T19:02:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone_platform.h: Cannot open: Input/output error 2023-12-19T19:02:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone04.c: Cannot open: Input/output error 2023-12-19T19:02:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone03.c: Cannot open: Input/output error 2023-12-19T19:02:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone06.c: Cannot open: Input/output error 2023-12-19T19:02:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone07.c: Cannot open: Input/output error 2023-12-19T19:02:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone05.c: Cannot open: Input/output error 2023-12-19T19:02:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone01.c: Cannot open: Input/output error 2023-12-19T19:02:22.361 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid: Cannot mkdir: Input/output error 2023-12-19T19:02:22.362 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.362 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/setresgid01.c: Cannot open: Input/output error 2023-12-19T19:02:22.362 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/setresgid02.c: Cannot open: Input/output error 2023-12-19T19:02:22.362 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/setresgid03.c: Cannot open: Input/output error 2023-12-19T19:02:22.362 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/brk: Cannot mkdir: Input/output error 2023-12-19T19:02:22.362 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/brk/brk01.c: Cannot open: Input/output error 2023-12-19T19:02:22.362 INFO:tasks.workunit.client.0.smithi059.stderr: 2023-12-19T19:02:22.362 INFO:tasks.workunit.client.0.smithi059.stderr:gzip: tar: ltp-full-20091231/testcases/kernel/syscalls/brk/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.363 INFO:tasks.workunit.client.0.smithi059.stderr:stdin: Stale file handle 2023-12-19T19:02:22.363 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid: Cannot mkdir: Input/output error 2023-12-19T19:02:22.363 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid/setreuid03.c: Cannot open: Input/output error 2023-12-19T19:02:22.363 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid/Makefile: Cannot open: Input/output error 2023-12-19T19:02:22.363 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid/setreuid07.c: Cannot open: Input/output error 2023-12-19T19:02:22.363 INFO:tasks.workunit.client.0.smithi059.stderr:tar: Unexpected EOF in archive 2023-12-19T19:02:22.363 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls: Cannot utime: Input/output error 2023-12-19T19:02:22.363 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases/kernel: Cannot utime: Input/output error 2023-12-19T19:02:22.363 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2023-12-19T19:02:22.364 INFO:tasks.workunit.client.0.smithi059.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2023-12-19T19:02:22.364 INFO:tasks.workunit.client.0.smithi059.stderr:tar: Error is not recoverable: exiting now 2023-12-19T19:02:22.364 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2023-12-19T19:02:22.364 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-19T19:02:22.443 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2023-12-19T19:02:22.444 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:02:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:22.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:22.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:22.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:22.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:22.575+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:23.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:22.774+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:23.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:22.774+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:23.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:22.774+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:23.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:22.774+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:22.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:23.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:22.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:23.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:22.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:23.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:22.846+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:23.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:23.824+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:23.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:23.824+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:23.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:23.824+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:23.828 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:23.826+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:23.829 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:23.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:23.829 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:23.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:23.829 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:23.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:23.829 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:23.593+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:23.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:23.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:23.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:23.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:23.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:23.733+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:23.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:23.734+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:24.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:24.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:24.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:24.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:24.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:24.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:24.747 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:24.638+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:24.744+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:24.744+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:24.744+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:25.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:24.745+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:24.779+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:24.779+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:24.779+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:24.780+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:25.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:25.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:25.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:25.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:25.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:25.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:25.780 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:25.632+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:25.789+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:26.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:25.789+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:26.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:25.789+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:26.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:25.791+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:26.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:25.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:26.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:25.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:26.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:25.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:26.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:25.778+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:26.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:26.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:26.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:26.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:26.833+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:26.678+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:26.678+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:26.678+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:26.679+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:26.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:26.809+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:26.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:26.809+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:26.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:26.809+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:26.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:26.810+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:27.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:27.104+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11665 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:27.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:27.790+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:27.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:27.790+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:27.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:27.790+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:27.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:27.790+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:27.875+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:27.875+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:27.875+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:27.877+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:27.636+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:27.636+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:27.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:27.636+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:27.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:27.637+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:28.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:28.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:28.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:28.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:28.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:28.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:28.928 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:28.601+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:28.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:28.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:28.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:28.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:28.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:28.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:28.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:28.830+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:28.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:28.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:28.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:28.927+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:29.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:29.622+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:29.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:29.622+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:29.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:29.622+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:29.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:29.623+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:29.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:29.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:29.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:29.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:29.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:29.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:29.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:29.860+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:29.929+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:30.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:29.929+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:30.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:29.929+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:30.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:29.929+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:30.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:30.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:30.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:30.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:30.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:30.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:30.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:30.577+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:30.955+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:30.955+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:31.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:30.955+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:31.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:30.956+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:31.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:30.903+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:31.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:30.903+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:31.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:30.903+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:31.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:30.903+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:31.937 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:31.870+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:31.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:31.870+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:31.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:31.870+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:31.938 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:31.871+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:31.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:31.622+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:31.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:31.622+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:31.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:31.622+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:31.938 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:31.623+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:32.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:32.105+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11759 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:31.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:31.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:31.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:32.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:31.936+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:32.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:32.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:32.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:32.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:32.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:32.898+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:32.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:32.899+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:32.667+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:32.667+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:32.667+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:32.668+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:33.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:32.967+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:33.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:32.967+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:33.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:32.967+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:33.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:32.969+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:33.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:33.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:33.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:33.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:33.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:33.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:33.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:33.660+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:34.019+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:34.019+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:34.019+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:34.019+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:34.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:33.946+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:34.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:33.946+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:34.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:33.946+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:34.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:33.947+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:34.902 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:34.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:34.902 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:34.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:34.902 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:34.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:34.902 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:34.629+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:35.013+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:35.013+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:35.013+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:35.014+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:35.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:34.899+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:35.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:34.899+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:35.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:34.899+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:35.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:34.900+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:35.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:35.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:35.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:35.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:35.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:35.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:35.941 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:35.581+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:35.938+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:35.938+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:35.938+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:35.939+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:36.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:36.038+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:36.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:36.038+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:36.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:36.038+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:36.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:36.039+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:36.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:36.578+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:36.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:36.578+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:36.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:36.578+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:36.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:36.579+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:37.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:37.107+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11857 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:37.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:37.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:37.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:37.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:37.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:37.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:37.055+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:37.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:36.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:37.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:36.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:37.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:36.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:37.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:36.931+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:37.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:37.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:37.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:37.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:37.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:37.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:37.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:37.601+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:38.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:38.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:38.096+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:38.097+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:37.881+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:37.881+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:37.881+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:37.882+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:38.839+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:38.839+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:38.839+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:38.839+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:38.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:38.602+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:38.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:38.602+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:38.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:38.602+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:38.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:38.603+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:39.057+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:39.057+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:39.057+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:39.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:39.058+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:39.802+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:39.802+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:39.802+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:39.803+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:39.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:39.615+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:39.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:39.615+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:39.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:39.615+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:39.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:39.616+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:40.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:40.050+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:40.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:40.050+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:40.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:40.050+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:40.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:40.051+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:40.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:40.623+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:40.878 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:40.623+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:40.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:40.623+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:40.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:40.624+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:40.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:40.837+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:40.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:40.837+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:40.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:40.837+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:40.879 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:40.837+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:41.090+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:41.090+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:41.090+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:41.091+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:41.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:41.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:41.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:41.663+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:41.810+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:41.810+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:41.810+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:41.810+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:42.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:42.108+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 11951 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:42.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:42.124+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:42.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:42.124+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:42.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:42.124+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:42.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:42.125+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:42.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:42.763+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:42.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:42.763+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:42.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:42.763+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:42.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:42.764+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:42.681+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:42.681+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:42.681+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:42.682+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:43.155+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:43.155+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:43.155+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:43.156+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:43.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:43.812+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:43.812+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:43.812+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:43.813+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:43.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:43.697+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:43.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:43.697+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:43.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:43.697+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:43.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:43.698+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:44.138+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:44.138+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:44.138+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:44.138+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:45.102 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:44.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:45.102 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:44.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:45.103 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:44.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:45.103 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:44.796+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:45.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:44.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:45.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:44.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:45.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:44.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:45.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:44.743+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:45.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:45.099+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:45.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:45.099+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:45.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:45.099+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:45.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:45.101+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:46.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:45.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:46.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:45.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:46.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:45.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:46.056 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:45.739+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:46.057 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:45.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:46.057 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:45.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:46.057 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:45.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:46.057 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:45.825+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:46.053+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:46.053+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:46.053+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:46.055+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:47.086 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:46.765+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:47.086 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:46.765+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:47.086 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:46.765+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:47.086 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:46.766+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:47.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:46.782+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:47.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:46.782+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:47.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:46.782+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:47.087 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:46.782+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:47.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:47.110+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12049 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:47.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:47.083+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:47.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:47.083+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:47.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:47.083+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:47.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:47.085+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:48.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:47.725+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:48.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:47.725+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:48.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:47.725+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:48.052 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:47.725+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:48.053 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:47.768+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:48.053 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:47.768+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:48.053 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:47.768+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:48.053 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:47.769+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:48.049+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:48.049+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:48.049+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:48.051+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:48.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:48.740+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:48.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:48.740+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:48.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:48.740+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:48.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:48.741+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:48.772+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:48.772+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:48.772+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:48.773+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:49.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:49.047+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:49.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:49.047+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:49.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:49.047+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:49.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:49.048+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:50.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:49.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:50.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:49.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:50.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:49.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:50.061 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:49.801+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:50.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:49.734+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:50.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:49.734+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:50.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:49.734+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:50.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:49.735+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:50.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:50.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:50.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:50.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:50.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:50.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:50.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:50.059+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:51.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:50.736+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:51.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:50.736+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:51.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:50.736+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:51.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:50.736+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:51.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:50.816+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:51.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:50.816+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:51.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:50.816+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:51.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:50.818+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:51.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:51.026+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:51.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:51.026+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:51.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:51.026+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:51.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:51.027+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:51.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:51.732+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:51.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:51.732+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:51.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:51.732+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:51.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:51.732+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:51.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:51.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:51.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:51.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:52.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:52.111+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12143 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:52.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:51.999+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:52.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:51.999+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:52.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:51.999+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:52.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:52.001+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:52.984 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:52.730+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:52.985 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:52.730+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:52.985 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:52.730+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:52.985 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:52.731+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:52.985 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:52.818+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:52.985 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:52.818+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:52.985 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:52.818+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:52.986 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:52.819+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:52.982+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:52.982+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:52.982+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:53.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:52.983+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:53.800 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:53.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:53.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:53.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:53.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:53.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:53.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:53.695+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:54.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:53.948+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:54.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:53.948+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:54.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:53.948+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:54.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:53.950+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:54.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:53.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:54.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:53.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:54.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:53.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:54.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:53.799+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:55.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:54.841+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:55.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:54.841+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:55.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:54.841+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:55.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:54.842+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:55.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:54.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:55.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:54.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:55.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:54.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:55.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:54.719+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:55.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:54.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:55.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:54.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:55.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:54.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:55.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:54.938+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:55.951+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:55.951+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:55.951+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:56.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:55.952+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:56.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:55.869+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:56.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:55.869+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:56.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:55.869+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:56.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:55.870+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:56.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:55.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:56.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:55.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:56.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:55.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:56.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:55.706+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:56.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:56.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:56.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:56.872+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:56.657+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:56.657+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:56.657+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:56.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:56.658+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:57.239 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:02:57.112+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12241 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:02:57.239 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:56.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:57.239 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:56.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:57.239 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:56.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:57.239 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:56.982+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:57.881+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:57.881+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:57.881+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:57.883+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:57.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:57.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:57.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:57.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:57.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:57.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:57.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:57.683+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:58.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:57.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:58.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:57.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:58.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:57.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:58.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:57.982+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:58.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:58.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:58.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:02:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:58.927+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:02:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:58.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:58.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:58.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:02:58.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:58.681+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:02:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:59.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:59.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:59.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:02:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:02:59.017+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:00.047 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:59.895+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:00.047 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:59.895+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:00.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:59.895+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:00.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:02:59.896+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:00.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:59.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:00.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:59.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:00.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:59.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:00.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:02:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:02:59.728+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:00.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:00.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:00.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:00.046+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:01.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:00.736+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:01.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:00.736+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:01.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:00.736+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:01.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:00.736+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:01.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:00.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:01.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:00.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:01.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:00.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:01.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:00.863+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:01.076+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:01.076+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:01.076+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:01.077+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:01.843+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:01.843+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:01.843+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:01.844+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:01.710+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:01.710+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:01.710+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:01.710+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:02.382 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:02.036+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:02.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:02.036+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:02.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:02.036+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:02.383 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:02.037+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:02.383 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:02.114+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12335 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:03.007 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:02.801+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:03.007 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:02.801+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:03.007 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:02.801+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:03.007 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:02.802+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:03.007 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:02.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:03.007 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:02.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:03.007 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:02.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:03.008 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:02.751+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:03.004+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:03.004+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:03.004+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:03.005+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:03.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:03.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:03.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:03.813+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:03.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:03.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:03.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:03.743+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:04.030+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:04.030+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:04.030+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:04.031+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:05.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:04.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:05.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:04.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:05.079 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:04.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:05.079 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:04.860+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:05.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:04.700+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:05.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:04.700+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:05.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:04.700+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:05.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:04.701+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:05.075+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:05.075+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:05.075+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:05.077+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:06.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:05.735+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:06.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:05.735+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:06.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:05.735+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:06.059 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:05.736+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:06.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:05.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:06.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:05.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:06.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:05.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:06.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:05.855+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:06.056+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:06.056+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:06.056+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:06.057+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:07.079 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:06.885+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:07.079 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:06.885+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:07.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:06.885+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:07.080 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:06.886+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:07.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:06.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:07.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:06.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:07.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:06.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:07.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:06.741+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:07.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:07.115+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12433 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:07.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:07.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:07.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:07.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:07.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:07.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:07.078+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:08.073 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:07.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:08.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:07.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:08.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:07.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:08.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:07.699+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:08.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:07.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:08.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:07.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:08.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:07.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:08.074 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:07.872+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:08.071+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:08.071+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:08.071+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:08.072+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:09.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:08.827+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:09.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:08.827+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:09.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:08.827+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:09.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:08.828+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:09.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:08.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:09.091 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:08.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:09.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:08.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:09.092 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:08.729+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:09.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:09.088+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:09.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:09.088+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:09.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:09.088+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:09.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:09.089+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:10.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:09.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:10.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:09.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:10.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:09.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:10.083 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:09.798+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:10.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:09.717+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:10.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:09.717+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:10.084 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:09.717+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:10.084 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:09.718+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:10.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:10.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:10.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:10.081+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:11.067 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:10.780+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:11.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:10.780+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:11.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:10.780+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:11.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:10.781+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:11.068 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:10.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:11.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:10.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:11.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:10.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:11.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:10.743+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:11.065+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:11.065+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:11.065+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:11.066+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:11.756+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:11.756+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:11.756+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:11.757+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:11.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:11.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:11.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:11.749+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:12.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:12.117+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12527 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:12.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:12.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:12.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:12.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:12.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:12.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:12.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:12.036+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:13.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:12.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:13.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:12.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:13.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:12.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:13.077 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:12.702+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:13.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:12.799+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:13.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:12.799+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:13.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:12.799+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:13.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:12.800+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:13.074+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:13.074+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:13.074+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:13.076+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:13.810+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:13.810+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:13.810+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:13.811+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:13.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:13.724+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:13.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:13.724+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:13.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:13.724+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:13.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:13.725+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:14.097+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:14.097+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:14.097+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:14.097+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:14.725+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:14.725+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:14.725+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:14.725+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:14.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:14.820+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:14.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:14.820+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:14.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:14.820+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:14.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:14.821+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:15.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:15.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:15.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:15.083+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:16.064 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:15.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:16.065 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:15.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:16.065 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:15.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:16.065 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:15.799+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:16.065 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:15.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:16.066 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:15.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:16.066 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:15.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:16.066 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:15.727+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:16.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:16.062+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:16.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:16.062+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:16.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:16.062+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:16.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:16.062+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:17.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:16.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:17.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:16.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:17.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:16.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:17.030 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:16.751+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:17.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:16.746+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:17.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:16.746+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:17.030 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:16.746+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:17.031 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:16.747+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:17.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:17.118+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12625 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:17.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:17.027+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:17.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:17.027+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:17.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:17.027+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:17.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:17.028+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:18.060 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:17.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:18.061 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:17.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:18.061 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:17.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:18.061 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:17.747+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:18.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:17.758+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:18.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:17.758+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:18.062 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:17.758+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:18.062 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:17.759+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:18.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:18.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:18.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:18.059+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:19.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:18.729+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:19.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:18.729+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:19.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:18.729+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:19.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:18.730+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:19.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:18.752+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:19.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:18.752+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:19.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:18.752+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:19.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:18.753+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:19.065+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:19.065+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:19.065+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:19.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:19.066+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:20.106 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:19.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:20.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:19.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:20.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:19.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:20.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:19.747+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:20.107 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:19.762+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:20.107 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:19.762+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:20.108 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:19.762+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:20.108 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:19.763+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:20.104+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:20.104+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:20.104+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:20.105+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:21.096 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:20.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:21.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:20.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:21.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:20.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:21.097 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:20.739+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:21.097 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:20.811+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:21.097 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:20.811+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:21.098 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:20.811+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:21.098 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:20.811+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:21.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:21.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:21.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:21.095+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:21.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:21.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:21.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:21.693+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:21.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:21.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:21.803+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:21.804+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:22.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:22.120+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12718 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:22.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:22.055+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:22.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:22.055+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:22.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:22.055+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:22.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:22.056+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:23.042 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:22.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:23.042 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:22.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:23.042 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:22.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:23.042 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:22.833+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:23.042 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:22.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:23.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:22.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:23.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:22.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:23.043 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:22.740+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:23.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:23.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:23.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:23.040+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:23.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:23.728+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:23.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:23.728+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:23.887 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:23.728+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:23.887 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:23.730+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:23.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:23.805+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:23.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:23.805+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:23.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:23.805+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:23.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:23.806+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:24.083+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:24.083+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:24.083+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:24.084+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:24.685+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:24.685+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:24.685+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:24.686+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:24.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:24.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:24.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:24.832+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:25.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:25.128+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:25.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:25.128+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:25.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:25.128+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:25.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:25.129+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:26.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:25.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:26.086 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:25.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:26.086 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:25.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:26.086 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:25.736+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:26.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:25.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:26.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:25.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:26.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:25.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:26.086 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:25.843+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:26.083+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:26.083+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:26.083+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:26.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:26.084+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:26.878+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:26.878+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:26.878+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:26.879+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:26.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:26.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:26.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:26.694+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:27.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:27.121+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12814 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:27.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:27.044+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:27.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:27.044+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:27.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:27.044+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:27.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:27.045+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:28.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:27.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:28.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:27.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:28.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:27.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:28.019 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:27.742+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:28.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:27.924+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:28.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:27.924+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:28.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:27.924+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:28.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:27.925+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:28.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:28.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:28.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:28.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:28.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:28.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:28.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:28.018+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:28.749+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:28.749+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:28.749+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:28.750+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:29.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:28.928+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:29.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:28.928+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:29.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:28.928+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:29.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:28.928+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:29.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:28.984+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:29.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:28.984+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:29.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:28.984+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:29.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:28.984+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:30.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:29.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:30.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:29.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:30.013 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:29.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:30.014 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:29.756+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:30.014 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:29.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:30.014 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:29.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:30.014 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:29.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:30.014 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:29.937+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:30.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:30.011+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:30.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:30.011+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:30.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:30.011+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:30.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:30.012+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:31.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:30.921+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:31.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:30.921+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:31.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:30.921+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:31.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:30.922+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:30.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:30.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:30.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:30.755+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:31.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:30.991+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:31.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:30.991+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:31.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:30.991+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:31.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:30.991+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:31.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:31.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:31.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:31.776+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:32.250 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:32.122+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 12908 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:32.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:31.955+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:32.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:31.955+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:32.250 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:31.955+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:32.251 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:31.956+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:32.251 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:32.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:32.251 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:32.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:32.251 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:32.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:32.251 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:32.036+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:33.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:32.806+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:33.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:32.806+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:33.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:32.806+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:33.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:32.807+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:33.103 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:33.004+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:33.104 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:33.004+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:33.104 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:33.004+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:33.104 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:33.005+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:33.104 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:32.922+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:33.104 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:32.922+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:33.104 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:32.922+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:33.105 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:32.923+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:33.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:33.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:33.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:33.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:33.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:33.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:33.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:33.812+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:33.956+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:33.956+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:33.956+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:33.957+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:34.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:34.034+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:34.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:34.034+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:34.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:34.034+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:34.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:34.035+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:34.918+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:34.918+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:34.918+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:35.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:34.919+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:34.988+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:34.988+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:34.988+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:34.989+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:35.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:34.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:35.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:34.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:35.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:34.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:35.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:34.775+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:36.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:35.795+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:36.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:35.795+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:36.175 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:35.795+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:36.175 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:35.796+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:36.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:35.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:36.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:35.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:36.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:35.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:36.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:35.959+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:36.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:35.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:36.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:35.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:36.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:35.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:36.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:35.880+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:37.126 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:36.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:37.126 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:36.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:37.126 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:36.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:37.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:36.813+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:37.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:36.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:37.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:36.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:37.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:36.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:37.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:36.925+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:37.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:36.881+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:37.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:36.881+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:37.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:36.881+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:37.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:36.882+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:37.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:37.124+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13005 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:37.789+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:37.789+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:37.789+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:37.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:37.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:37.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:37.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:37.909+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:37.852+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:37.852+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:38.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:37.852+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:38.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:37.852+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:38.888+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:38.888+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:38.888+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:39.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:38.888+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:38.904+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:38.904+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:38.904+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:38.905+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:38.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:38.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:38.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:39.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:38.751+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:40.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:39.723+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:40.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:39.723+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:40.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:39.723+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:40.152 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:39.724+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:40.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:39.879+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:40.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:39.879+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:40.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:39.879+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:40.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:39.880+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:40.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:39.931+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:40.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:39.931+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:40.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:39.931+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:40.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:39.932+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:40.928+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:40.928+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:40.928+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:41.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:40.929+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:41.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:40.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:41.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:40.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:41.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:40.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:41.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:40.747+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:41.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:40.922+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:41.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:40.922+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:41.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:40.922+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:41.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:40.923+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:41.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:41.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:41.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:41.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:41.897+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:41.744+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:41.744+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:41.744+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:41.745+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:41.899+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:41.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:41.899+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:41.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:41.899+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:41.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:41.899+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:42.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:42.125+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13100 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:42.865+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:42.865+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:42.865+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:42.866+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:43.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:42.883+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:43.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:42.883+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:43.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:42.883+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:43.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:42.884+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:43.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:42.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:43.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:42.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:43.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:42.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:43.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:42.787+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:43.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:43.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:43.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:43.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:43.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:43.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:43.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:43.763+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:43.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:43.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:43.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:43.890+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:44.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:43.872+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:44.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:43.872+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:44.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:43.872+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:44.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:43.873+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:44.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:44.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:44.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:45.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:44.843+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:44.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:44.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:44.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:44.909+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:45.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:44.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:45.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:44.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:45.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:44.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:45.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:44.775+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:45.806+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:45.806+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:45.806+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:45.807+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:45.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:45.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:45.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:45.910+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:46.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:45.815+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:46.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:45.815+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:46.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:45.815+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:46.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:45.815+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:47.129 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:46.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:47.129 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:46.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:47.130 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:46.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:47.130 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:46.786+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:47.130 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:46.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:47.130 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:46.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:47.130 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:46.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:47.130 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:46.784+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:47.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:46.904+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:47.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:46.904+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:47.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:46.904+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:47.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:46.905+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:47.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:47.126+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13196 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:47.914+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:47.914+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:47.914+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:47.915+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:47.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:47.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:47.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:47.761+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:48.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:47.820+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:48.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:47.820+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:48.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:47.820+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:48.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:47.821+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:48.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:48.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:48.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:48.772+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:48.861+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:48.861+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:48.861+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:48.861+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:49.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:48.871+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:49.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:48.871+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:49.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:48.871+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:49.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:48.872+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:50.152 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:49.855+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:50.152 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:49.855+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:50.152 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:49.855+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:50.152 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:49.855+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:50.153 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:49.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:50.153 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:49.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:50.153 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:49.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:50.153 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:49.889+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:50.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:49.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:50.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:49.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:50.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:49.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:50.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:49.733+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:51.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:50.698+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:51.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:50.698+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:51.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:50.698+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:51.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:50.699+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:51.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:50.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:51.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:50.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:51.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:50.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:51.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:50.865+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:51.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:50.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:51.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:50.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:51.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:50.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:51.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:50.815+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:51.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:51.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:51.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:51.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:51.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:51.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:51.833 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:51.688+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:52.130 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:51.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:52.130 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:51.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:52.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:51.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:52.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:51.831+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:52.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:51.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:52.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:51.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:52.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:51.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:52.131 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:51.831+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:52.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:52.128+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13290 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:52.853+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:52.853+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:52.853+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:52.853+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:52.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:52.855+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:52.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:52.855+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:52.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:52.855+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:52.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:52.856+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:52.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:52.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:52.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:52.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:52.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:52.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:52.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:52.642+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:53.852 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:53.675+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:53.853 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:53.675+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:53.853 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:53.675+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:53.853 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:53.677+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:53.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:53.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:53.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:53.898+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:54.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:53.868+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:54.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:53.868+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:54.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:53.868+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:54.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:53.868+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:54.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:54.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:54.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:54.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:54.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:54.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:54.862 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:54.645+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:54.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:54.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:54.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:54.860+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:55.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:54.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:55.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:54.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:55.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:54.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:55.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:54.903+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:55.885+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:55.885+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:55.885+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:55.887+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:55.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:55.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:55.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:55.931+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:55.691+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:55.691+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:55.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:55.691+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:55.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:55.692+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:56.650+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:56.650+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:56.650+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:56.651+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:56.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:56.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:56.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:56.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:56.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:56.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:56.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:56.873+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:57.257 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:03:57.129+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13386 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:03:57.257 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:56.975+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:57.257 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:56.975+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:57.257 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:56.975+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:57.258 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:56.976+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:57.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:57.612+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:57.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:57.612+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:57.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:57.612+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:57.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:57.614+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:57.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:57.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:57.903+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:57.904+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:58.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:57.931+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:58.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:57.931+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:58.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:57.931+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:58.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:57.932+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:58.885 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:58.604+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:58.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:58.604+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:58.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:58.604+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:58.886 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:58.605+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:03:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:58.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:58.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:58.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:03:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:58.884+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:59.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:58.883+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:59.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:58.883+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:59.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:58.883+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:03:59.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:58.884+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:03:59.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:59.623+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:59.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:59.623+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:59.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:59.623+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:03:59.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:03:59.624+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:00.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:59.911+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:00.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:59.911+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:00.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:59.911+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:00.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:03:59.912+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:00.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:59.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:00.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:59.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:00.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:59.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:00.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:03:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:03:59.923+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:00.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:00.879+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:00.882 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:00.879+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:00.882 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:00.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:00.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:00.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:00.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:00.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:00.883 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:00.631+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:00.879+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:00.880+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:01.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:00.973+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:01.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:00.973+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:01.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:00.973+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:01.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:00.974+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:01.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:01.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:01.905 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:01.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:01.906 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:01.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:01.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:01.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:01.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:01.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:01.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:01.653+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:01.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:01.653+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:02.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:01.903+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:02.201 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:02.131+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13481 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:02.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:02.007+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:02.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:02.007+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:02.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:02.007+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:02.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:02.008+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:02.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:02.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:02.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:02.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:02.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:02.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:02.884 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:02.615+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:02.881+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:02.881+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:02.881+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:02.882+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:03.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:03.035+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:03.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:03.035+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:03.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:03.035+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:03.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:03.035+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:03.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:03.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:03.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:03.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:03.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:03.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:03.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:03.597+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:04.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:03.873+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:04.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:03.873+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:04.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:03.873+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:04.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:03.874+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:04.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:04.084+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:04.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:04.084+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:04.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:04.084+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:04.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:04.085+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:04.851+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:04.851+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:04.851+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:04.852+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:04.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:04.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:04.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:04.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:04.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:04.621+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:04.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:04.622+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:05.117+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:05.117+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:05.117+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:05.117+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:05.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:05.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:05.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:05.873+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:05.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:05.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:05.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:05.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:05.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:05.639+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:05.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:05.640+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:06.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:06.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:06.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:06.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:06.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:06.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:06.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:06.156+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:06.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:06.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:06.656+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:06.657+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:06.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:06.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:06.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:06.833+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:07.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:07.132+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13578 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:07.198+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:07.198+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:07.198+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:07.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:07.199+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:07.869+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:07.869+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:07.869+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:07.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:07.870+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:07.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:07.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:07.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:07.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:07.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:07.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:07.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:07.663+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:08.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:08.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:08.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:08.236+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:09.190 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:08.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:09.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:08.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:09.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:08.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:09.191 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:08.916+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:09.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:08.707+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:09.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:08.707+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:09.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:08.707+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:09.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:08.708+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:09.192 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:09.188+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:09.192 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:09.188+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:09.192 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:09.188+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:09.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:09.188+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:10.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:09.949+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:10.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:09.949+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:10.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:09.949+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:10.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:09.949+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:10.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:09.709+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:10.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:09.709+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:10.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:09.709+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:10.155 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:09.709+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:10.192+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:10.192+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:10.192+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:10.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:10.192+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:11.152 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:10.992+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:11.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:10.992+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:11.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:10.992+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:11.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:10.993+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:11.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:10.757+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:11.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:10.757+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:11.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:10.757+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:11.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:10.757+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:11.150+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:11.150+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:11.150+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:11.150+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:11.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:11.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:11.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:11.780+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:12.259 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:12.134+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13673 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:12.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:11.989+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:12.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:11.989+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:12.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:11.989+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:12.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:11.989+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:12.260 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:12.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:12.260 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:12.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:12.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:12.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:12.261 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:12.200+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:13.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:12.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:13.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:12.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:13.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:12.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:13.025 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:12.756+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:13.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:13.220+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:13.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:13.220+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:13.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:13.220+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:13.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:13.221+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:13.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:13.022+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:13.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:13.022+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:13.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:13.022+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:13.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:13.023+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:13.796+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:13.796+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:13.796+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:13.797+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:14.012+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:14.012+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:14.012+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:14.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:14.014+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:14.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:14.251+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:14.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:14.251+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:14.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:14.252+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:14.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:14.252+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:14.995+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:14.995+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:14.995+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:14.995+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:15.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:14.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:15.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:14.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:15.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:14.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:15.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:14.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:15.250+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:15.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:15.250+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:15.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:15.250+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:15.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:15.251+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:16.110 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:15.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:16.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:15.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:16.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:15.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:16.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:15.762+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:16.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:15.953+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:16.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:15.953+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:16.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:15.953+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:16.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:15.954+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:16.271+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:16.271+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:16.271+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:16.271+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:17.138 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:16.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:17.138 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:16.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:17.138 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:16.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:17.138 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:16.756+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:17.138 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:16.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:17.138 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:16.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:17.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:16.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:17.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:16.951+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:17.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:17.135+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13770 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:17.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:17.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:17.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:17.249+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:18.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:17.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:18.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:17.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:18.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:17.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:18.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:17.803+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:18.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:17.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:18.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:17.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:18.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:17.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:18.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:17.909+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:18.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:18.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:18.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:18.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:18.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:18.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:18.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:18.266+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:18.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:18.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:18.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:18.884+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:19.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:18.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:19.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:18.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:19.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:18.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:19.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:18.771+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:19.239+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:19.239+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:19.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:19.239+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:19.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:19.240+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:20.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:19.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:20.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:19.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:20.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:19.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:20.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:19.769+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:20.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:19.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:20.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:19.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:20.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:19.878+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:20.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:19.879+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:20.280+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:20.280+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:20.280+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:20.281+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:20.807+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:20.807+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:20.807+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:20.809+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:21.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:20.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:21.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:20.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:21.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:20.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:21.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:20.833+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:21.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:21.306+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:21.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:21.306+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:21.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:21.306+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:21.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:21.306+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:21.873+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:21.873+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:21.873+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:21.874+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:21.858+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:21.858+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:21.858+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:21.858+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:22.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:22.136+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13865 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:22.278+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:22.278+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:22.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:22.278+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:22.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:22.278+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:22.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:22.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:22.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:22.908+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:22.901+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:22.901+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:22.901+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:22.902+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:23.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:23.249+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:23.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:23.249+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:23.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:23.249+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:23.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:23.250+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:24.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:23.875+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:24.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:23.875+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:24.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:23.875+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:24.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:23.876+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:24.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:23.860+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:24.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:23.860+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:24.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:23.860+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:24.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:23.861+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:24.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:24.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:24.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:24.267+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:24.853+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:24.853+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:24.853+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:24.854+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:24.914+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:24.914+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:24.914+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:24.915+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:25.230+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:25.230+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:25.230+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:25.231+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:25.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:25.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:25.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:25.882+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:26.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:25.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:26.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:25.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:26.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:25.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:26.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:25.932+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:26.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:26.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:26.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:26.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:26.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:26.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:26.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:26.200+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:27.200 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:27.138+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 13962 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:27.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:26.930+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:27.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:26.930+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:27.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:26.930+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:27.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:26.932+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:27.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:26.925+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:27.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:26.925+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:27.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:26.925+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:27.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:26.925+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:27.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:27.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:27.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:27.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:27.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:27.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:27.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:27.241+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:27.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:27.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:27.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:28.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:27.959+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:28.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:27.898+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:28.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:27.898+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:28.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:27.898+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:28.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:27.899+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:28.249+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:28.249+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:28.249+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:28.250+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:29.004+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:29.004+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:29.004+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:29.004+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:29.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:28.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:29.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:28.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:29.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:28.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:29.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:28.872+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:29.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:29.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:29.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:29.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:29.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:29.244+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:30.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:29.969+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:30.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:29.969+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:30.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:29.969+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:30.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:29.969+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:30.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:29.870+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:30.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:29.870+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:30.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:29.870+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:30.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:29.871+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:30.255+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:30.255+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:30.255+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:30.256+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:30.996+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:30.996+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:30.996+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:30.997+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:30.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:30.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:30.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:30.861+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:31.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:31.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:31.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:31.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:31.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:31.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:31.686 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:31.234+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:31.842+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:31.842+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:31.842+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:31.842+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:32.214 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:32.139+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14057 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:32.214 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:31.957+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:32.214 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:31.957+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:32.214 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:31.957+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:32.215 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:31.958+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:32.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:32.211+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:32.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:32.211+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:32.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:32.211+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:32.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:32.212+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:33.173 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:32.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:33.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:32.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:33.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:32.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:33.174 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:32.938+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:33.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:32.814+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:33.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:32.814+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:33.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:32.814+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:33.175 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:32.815+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:33.171+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:33.171+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:33.171+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:33.172+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:33.923+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:33.923+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:33.923+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:33.924+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:34.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:33.856+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:34.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:33.856+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:34.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:33.856+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:34.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:33.857+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:34.542 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:34.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:34.542 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:34.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:34.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:34.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:34.543 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:34.209+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:34.959+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:34.959+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:34.959+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:34.959+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:35.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:34.809+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:35.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:34.809+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:35.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:34.809+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:35.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:34.810+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:35.197+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:35.197+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:35.197+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:35.198+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:36.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:35.829+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:36.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:35.829+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:36.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:35.829+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:36.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:35.829+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:36.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:35.938+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:36.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:35.938+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:36.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:35.938+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:36.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:35.939+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:36.216+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:36.216+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:36.216+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:36.216+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:37.143 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:36.841+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:37.143 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:36.841+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:37.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:36.841+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:37.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:36.843+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:37.144 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:36.893+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:37.144 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:36.893+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:37.144 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:36.893+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:37.144 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:36.894+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:37.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:37.140+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14154 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:37.263+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:37.263+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:37.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:37.263+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:37.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:37.263+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:37.839+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:37.839+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:37.839+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:38.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:37.840+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:37.873+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:37.873+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:37.873+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:37.873+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:38.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:38.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:38.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:38.243+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:38.801+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:38.801+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:38.801+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:38.802+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:38.850+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:38.850+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:38.850+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:38.851+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:39.292+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:39.292+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:39.292+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:39.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:39.292+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:40.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:39.852+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:40.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:39.852+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:40.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:39.852+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:40.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:39.853+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:40.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:39.831+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:40.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:39.831+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:40.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:39.831+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:40.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:39.833+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:40.310+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:40.310+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:40.310+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:40.310+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:40.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:40.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:40.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:40.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:40.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:40.881+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:40.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:40.840+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:40.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:40.840+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:40.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:40.840+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:40.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:40.841+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:41.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:41.284+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:41.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:41.284+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:41.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:41.284+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:41.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:41.285+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:41.853+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:41.853+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:41.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:41.854+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:41.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:41.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:41.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:41.860+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:42.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:42.142+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14249 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:42.245+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:42.245+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:42.245+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:42.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:42.245+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:42.838+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:42.838+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:42.838+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:42.840+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:43.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:42.822+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:43.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:42.822+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:43.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:42.822+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:43.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:42.822+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:43.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:43.228+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:43.587 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:43.228+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:43.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:43.228+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:43.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:43.228+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:43.872 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:43.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:43.872 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:43.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:43.872 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:43.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:43.872 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:43.787+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:43.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:43.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:43.870+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:43.870+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:44.260+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:44.260+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:44.260+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:44.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:44.260+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:44.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:44.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:44.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:44.776+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:44.881+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:44.881+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:44.881+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:44.882+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:45.268+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:45.268+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:45.268+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:45.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:45.268+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:45.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:45.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:45.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:45.768+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:45.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:45.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:45.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:45.883+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:46.285+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:46.285+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:46.285+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:46.286+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:47.147 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:46.911+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:47.147 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:46.911+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:47.147 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:46.911+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:47.147 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:46.912+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:47.147 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:46.813+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:47.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:46.813+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:47.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:46.813+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:47.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:46.814+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:47.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:47.144+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14347 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:47.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:47.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:47.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:47.248+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:47.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:47.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:47.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:48.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:47.951+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:47.794+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:47.794+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:47.794+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:47.796+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:48.227+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:48.227+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:48.227+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:48.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:48.228+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:48.833+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:48.833+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:48.833+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:48.834+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:49.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:48.914+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:49.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:48.914+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:49.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:48.914+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:49.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:48.916+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:49.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:49.270+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:49.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:49.270+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:49.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:49.270+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:49.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:49.270+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:50.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:49.867+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:50.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:49.867+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:50.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:49.867+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:50.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:49.868+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:50.157 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:49.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:50.157 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:49.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:50.157 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:49.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:50.157 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:49.792+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:50.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:50.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:50.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:50.235+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:51.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:50.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:51.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:50.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:51.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:50.832+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:51.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:50.833+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:51.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:50.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:51.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:50.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:51.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:50.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:51.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:50.780+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:51.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:51.264+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:51.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:51.264+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:51.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:51.264+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:51.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:51.265+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:51.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:51.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:51.791+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:51.792+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:51.748+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:51.748+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:51.748+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:51.748+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:52.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:52.146+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14437 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:52.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:52.253+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:52.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:52.253+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:52.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:52.253+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:52.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:52.253+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:52.779+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:52.779+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:52.779+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:52.780+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:53.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:52.769+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:53.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:52.769+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:53.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:52.769+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:53.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:52.770+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:53.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:53.258+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:53.258+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:53.258+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:53.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:53.258+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:53.781+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:53.781+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:53.781+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:53.781+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:53.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:53.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:53.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:53.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:53.770+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:54.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:54.224+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:54.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:54.224+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:54.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:54.224+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:54.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:54.225+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:55.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:54.776+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:55.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:54.776+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:55.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:54.776+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:55.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:54.777+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:55.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:54.780+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:55.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:54.780+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:55.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:54.780+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:55.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:54.781+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:55.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:55.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:55.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:55.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:55.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:55.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:55.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:55.243+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:56.046 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:55.808+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:56.047 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:55.808+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:56.047 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:55.808+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:56.047 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:55.809+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:56.047 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:55.783+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:56.047 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:55.783+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:56.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:55.783+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:56.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:55.784+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:56.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:56.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:56.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:56.209+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:57.150 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:56.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:57.150 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:56.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:57.151 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:56.803+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:57.151 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:56.804+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:57.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:56.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:57.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:56.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:57.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:56.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:57.151 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:56.824+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:57.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:04:57.148+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14527 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:04:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:57.180+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:57.180+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:57.180+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:57.181+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:57.756+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:57.756+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:58.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:57.756+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:58.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:57.757+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:58.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:57.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:58.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:57.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:58.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:57.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:58.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:57.779+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:58.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:58.214+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:58.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:58.214+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:58.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:58.214+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:58.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:58.214+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:58.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:58.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:59.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:58.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:04:59.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:58.758+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:04:59.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:58.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:59.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:58.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:59.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:58.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:04:59.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:58.761+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:04:59.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:59.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:59.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:59.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:59.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:59.234+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:04:59.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:04:59.235+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:00.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:59.720+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:00.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:59.720+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:00.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:59.720+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:00.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:04:59.721+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:00.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:59.739+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:00.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:59.739+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:00.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:59.739+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:00.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:04:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:04:59.740+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:00.211+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:00.211+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:00.211+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:00.211+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:01.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:00.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:01.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:00.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:01.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:00.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:01.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:00.759+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:01.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:00.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:01.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:00.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:01.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:00.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:01.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:00.739+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:01.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:01.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:01.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:01.167+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:01.771+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:01.771+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:01.771+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:01.772+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:01.712+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:01.712+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:01.712+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:01.713+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:02.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:02.149+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14615 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:02.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:02.204+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:02.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:02.204+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:02.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:02.204+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:02.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:02.205+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:02.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:02.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:02.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:02.745+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:02.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:02.724+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:02.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:02.724+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:02.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:02.724+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:02.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:02.725+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:03.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:03.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:03.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:03.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:03.243+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:03.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:03.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:03.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:03.716+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:03.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:03.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:03.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:03.746+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:04.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:04.207+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:04.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:04.207+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:04.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:04.207+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:04.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:04.207+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:04.743+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:04.743+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:04.743+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:05.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:04.744+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:05.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:04.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:05.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:04.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:05.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:04.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:05.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:04.793+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:05.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:05.250+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:05.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:05.250+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:05.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:05.250+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:05.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:05.251+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:06.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:05.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:06.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:05.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:06.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:05.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:05.804+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:06.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:05.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:06.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:05.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:06.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:05.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:06.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:05.729+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:06.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:06.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:06.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:06.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:06.237+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:07.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:06.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:07.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:06.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:07.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:06.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:07.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:06.724+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:07.155 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:06.828+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:07.155 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:06.828+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:07.155 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:06.828+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:07.155 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:06.829+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:07.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:07.151+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14704 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:07.238+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:07.238+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:07.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:07.238+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:07.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:07.239+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:07.734+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:07.734+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:07.734+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:07.736+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:08.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:07.840+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:08.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:07.840+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:08.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:07.840+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:08.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:07.841+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:08.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:08.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:08.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:08.202+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:08.773+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:08.773+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:08.773+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:08.774+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:08.890+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:08.890+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:08.890+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:08.892+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:09.239+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:09.239+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:09.239+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:09.240+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:10.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:09.767+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:10.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:09.767+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:10.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:09.767+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:10.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:09.768+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:10.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:09.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:10.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:09.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:10.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:09.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:10.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:09.850+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:10.282+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:10.282+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:10.282+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:10.282+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:10.741+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:10.741+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:10.741+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:10.742+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:11.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:10.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:11.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:10.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:11.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:10.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:11.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:10.803+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:11.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:11.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:11.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:11.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:11.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:11.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:11.689 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:11.244+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:11.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:11.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:11.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:11.783+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:11.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:11.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:11.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:11.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:11.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:11.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:11.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:11.707+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:12.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:12.152+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14792 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:12.222+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:12.222+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:12.222+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:12.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:12.222+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:12.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:12.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:12.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:13.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:12.794+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:13.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:12.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:13.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:12.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:13.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:12.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:13.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:12.724+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:13.217+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:13.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:13.217+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:13.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:13.217+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:13.590 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:13.218+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:13.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:13.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:13.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:13.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:13.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:13.721+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:13.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:13.722+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:13.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:13.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:13.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:13.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:13.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:13.821+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:13.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:13.823+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:14.188+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:14.188+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:14.188+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:14.188+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:15.189 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:14.736+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:15.189 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:14.736+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:15.189 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:14.736+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:15.189 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:14.737+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:15.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:14.806+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:15.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:14.806+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:15.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:14.806+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:15.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:14.806+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:15.190 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:15.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:15.190 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:15.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:15.190 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:15.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:15.187+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:16.189 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:15.737+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:16.190 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:15.737+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:16.190 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:15.737+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:16.190 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:15.738+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:16.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:15.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:16.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:15.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:16.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:15.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:16.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:15.823+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:16.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:16.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:16.183+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:16.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:16.183+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:17.156 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:16.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:17.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:16.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:17.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:16.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:17.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:16.711+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:17.157 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:16.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:17.157 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:16.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:17.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:16.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:17.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:16.780+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:17.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:17.154+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14883 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:17.163+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:17.163+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:17.163+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:17.163+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:18.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:17.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:18.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:17.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:18.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:17.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:18.185 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:17.746+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:18.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:17.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:18.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:17.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:18.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:17.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:18.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:17.742+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:18.182+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:18.182+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:18.182+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:18.183+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:18.712+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:18.712+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:18.712+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:18.713+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:19.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:18.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:19.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:18.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:19.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:18.784+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:19.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:18.784+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:19.229+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:19.229+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:19.229+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:19.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:19.230+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:20.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:19.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:20.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:19.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:20.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:19.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:20.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:19.760+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:20.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:19.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:20.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:19.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:20.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:19.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:20.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:19.721+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:20.245+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:20.245+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:20.245+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:20.246+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:20.800+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:20.800+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:20.800+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:20.801+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:20.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:20.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:20.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:20.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:20.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:20.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:20.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:20.675+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:21.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:21.241+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:21.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:21.241+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:21.639 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:21.241+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:21.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:21.242+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:21.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:21.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:21.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:21.776+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:21.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:21.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:21.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:21.638+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:22.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:22.155+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 14972 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:22.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:22.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:22.240+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:22.240+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:22.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:22.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:22.745+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:22.746+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:22.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:22.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:22.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:22.621+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:23.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:23.232+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:23.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:23.232+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:23.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:23.232+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:23.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:23.233+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:23.582+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:23.582+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:23.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:23.582+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:23.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:23.583+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:23.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:23.790+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:23.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:23.790+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:23.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:23.790+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:23.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:23.791+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:24.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:24.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:24.606 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:24.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:24.606 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:24.248+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:24.606 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:24.248+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:24.603+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:24.603+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:24.603+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:24.604+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:24.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:24.768+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:24.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:24.768+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:24.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:24.768+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:24.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:24.769+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:25.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:25.232+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:25.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:25.232+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:25.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:25.232+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:25.625 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:25.233+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:25.737+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:25.737+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:25.737+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:25.738+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:25.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:25.622+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:25.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:25.622+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:25.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:25.622+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:25.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:25.623+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:26.646 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:26.206+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:26.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:26.206+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:26.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:26.206+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:26.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:26.207+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:26.702+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:26.702+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:26.702+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:26.702+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:26.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:26.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:26.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:26.645+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:27.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:27.157+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15062 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:27.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:27.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:27.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:27.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:27.168+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:27.747+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:27.747+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:27.747+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:27.748+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:27.660+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:27.660+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:27.660+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:27.661+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:28.149+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:28.149+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:28.149+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:28.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:28.150+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:28.725+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:28.725+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:28.725+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:28.726+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:28.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:28.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:28.663+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:28.664+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:29.138+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:29.138+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:29.138+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:29.139+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:29.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:29.698+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:29.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:29.698+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:29.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:29.698+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:29.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:29.699+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:29.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:29.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:29.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:29.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:29.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:29.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:29.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:29.673+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:30.109+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:30.109+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:30.109+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:30.109+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:31.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:30.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:31.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:30.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:31.120 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:30.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:31.121 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:30.729+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:31.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:30.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:31.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:30.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:31.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:30.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:31.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:30.708+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:31.118+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:31.118+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:31.118+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:31.118+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:31.688+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:32.015 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:31.688+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:32.015 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:31.688+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:32.016 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:31.689+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:32.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:31.702+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:32.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:31.702+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:32.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:31.702+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:32.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:31.702+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:32.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:32.159+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15151 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:32.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:32.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:32.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:32.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:32.155+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:32.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:32.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:32.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:32.724+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:32.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:32.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:32.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:32.675+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:33.180+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:33.180+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:33.180+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:33.181+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:33.687+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:33.687+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:33.687+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:33.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:33.688+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:33.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:33.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:33.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:33.631+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:34.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:34.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:34.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:34.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:34.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:34.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:34.206 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:34.202+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:34.685+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:34.685+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:34.685+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:34.686+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:34.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:34.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:34.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:34.659+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:35.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:35.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:35.654 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:35.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:35.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:35.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:35.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:35.200+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:35.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:35.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:35.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:35.652+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:35.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:35.734+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:35.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:35.734+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:35.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:35.734+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:35.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:35.736+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:36.227+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:36.227+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:36.227+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:36.228+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:36.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:36.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:36.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:36.672+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:36.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:36.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:36.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:36.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:36.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:36.759+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:36.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:36.760+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:37.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:37.160+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15242 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:37.215+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:37.215+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:37.215+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:37.216+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:37.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:37.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:37.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:37.694+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:37.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:37.789+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:37.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:37.789+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:37.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:37.789+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:37.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:37.790+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:38.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:38.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:38.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:38.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:38.248+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:38.749+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:38.749+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:38.749+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:38.750+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:38.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:38.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:38.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:38.740+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:39.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:39.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:39.202+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:39.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:39.203+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:40.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:39.735+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:40.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:39.735+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:40.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:39.735+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:40.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:39.735+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:40.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:39.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:40.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:39.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:40.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:39.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:40.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:39.774+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:40.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:40.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:40.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:40.156+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:41.142 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:40.714+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:41.143 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:40.714+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:41.143 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:40.714+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:41.143 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:40.715+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:41.143 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:40.799+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:41.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:40.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:41.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:40.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:41.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:40.801+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:41.140+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:41.140+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:41.140+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:41.141+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:41.748+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:41.748+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:41.748+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:41.749+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:41.751+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:41.751+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:41.751+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:41.752+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:42.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:42.162+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15331 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:42.112+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:42.112+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:42.112+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:42.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:42.112+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:43.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:42.739+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:43.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:42.739+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:43.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:42.739+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:43.067 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:42.740+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:43.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:42.703+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:43.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:42.703+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:43.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:42.703+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:43.068 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:42.704+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:43.064+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:43.064+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:43.064+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:43.065+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:43.783+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:43.783+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:43.783+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:43.783+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:43.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:43.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:43.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:43.746+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:44.107+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:44.107+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:44.107+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:44.107+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:45.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:44.777+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:45.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:44.777+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:45.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:44.777+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:45.155 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:44.778+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:45.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:44.765+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:45.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:44.765+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:45.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:44.765+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:45.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:44.766+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:45.152+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:45.152+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:45.152+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:45.153+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:46.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:45.818+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:46.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:45.818+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:46.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:45.818+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:46.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:45.818+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:46.180 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:45.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:46.180 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:45.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:46.180 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:45.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:46.180 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:45.775+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:46.176+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:46.176+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:46.176+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:46.177+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:47.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:46.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:47.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:46.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:47.166 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:46.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:47.167 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:46.756+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:47.167 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:46.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:47.167 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:46.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:47.167 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:46.775+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:47.167 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:46.776+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:47.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:47.163+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15421 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:47.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:47.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:47.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:47.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:47.200+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:48.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:47.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:48.171 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:47.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:48.172 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:47.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:48.172 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:47.770+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:48.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:47.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:48.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:47.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:48.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:47.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:48.172 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:47.758+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:48.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:48.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:48.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:48.169+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:49.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:48.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:49.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:48.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:49.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:48.797+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:49.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:48.798+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:48.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:48.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:49.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:48.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:49.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:48.776+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:49.196+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:49.196+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:49.196+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:49.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:49.197+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:50.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:49.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:50.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:49.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:50.157 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:49.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:50.157 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:49.755+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:50.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:49.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:50.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:49.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:50.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:49.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:50.157 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:49.758+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:50.154+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:50.154+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:50.154+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:50.154+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:51.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:50.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:51.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:50.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:51.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:50.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:51.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:50.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:51.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:50.755+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:51.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:50.755+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:51.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:50.755+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:51.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:50.755+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:51.195+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:51.195+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:51.195+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:51.196+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:51.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:51.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:51.723+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:51.723+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:51.810+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:51.810+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:51.810+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:51.810+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:52.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:52.165+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15510 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:52.456 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:52.181+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:52.457 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:52.181+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:52.457 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:52.181+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:52.457 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:52.181+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:52.719+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:52.719+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:52.719+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:52.720+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:53.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:52.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:53.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:52.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:53.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:52.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:53.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:52.813+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:53.592 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:53.200+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:53.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:53.200+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:53.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:53.200+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:53.593 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:53.200+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:53.672+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:53.672+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:53.672+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:53.673+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:53.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:53.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:53.792+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:53.793+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:54.181+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:54.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:54.181+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:54.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:54.181+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:54.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:54.182+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:54.630+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:54.630+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:54.630+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:54.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:54.631+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:54.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:54.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:54.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:54.777+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:55.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:55.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:55.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:55.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:55.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:55.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:55.606 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:55.201+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:55.603+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:55.603+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:55.603+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:55.603+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:55.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:55.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:55.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:55.742+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:56.158+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:56.158+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:56.158+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:56.158+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:56.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:56.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:56.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:56.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:56.575+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:56.731+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:56.731+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:56.731+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:56.732+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:57.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:05:57.167+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15601 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:05:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:57.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:57.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:57.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:57.121+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:57.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:57.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:57.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:57.567+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:57.777+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:57.777+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:57.777+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:57.778+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:58.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:58.097+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:58.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:58.097+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:58.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:58.097+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:58.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:58.097+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:58.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:58.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:58.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:58.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:58.585+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:58.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:58.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:58.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:58.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:58.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:58.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:58.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:58.774+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:05:59.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:59.063+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:59.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:59.063+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:59.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:59.063+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:05:59.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:05:59.063+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:59.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:59.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:59.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:59.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:05:59.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:05:59.619+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:05:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:59.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:59.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:59.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:05:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:05:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:05:59.824+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:00.074+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:00.074+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:00.074+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:00.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:00.074+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:00.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:00.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:00.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:00.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:00.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:00.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:00.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:00.780+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:00.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:00.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:00.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:00.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:00.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:00.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:00.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:00.581+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:01.047+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:01.047+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:01.047+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:01.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:01.048+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:01.737+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:01.737+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:01.737+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:01.738+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:01.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:01.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:01.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:01.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:01.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:01.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:01.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:01.597+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:02.292 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:02.168+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15689 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:02.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:02.020+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:02.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:02.020+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:02.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:02.020+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:02.293 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:02.021+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:02.605+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:02.605+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:02.605+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:02.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:02.606+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:02.721+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:02.721+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:02.721+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:02.722+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:03.008+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:03.008+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:03.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:03.008+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:03.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:03.008+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:03.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:03.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:03.682+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:03.682+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:03.649+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:03.649+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:03.649+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:03.650+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:04.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:04.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:04.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:04.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:04.033+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:04.666+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:04.666+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:04.666+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:04.667+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:04.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:04.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:04.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:04.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:04.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:04.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:04.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:04.608+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:04.997+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:04.997+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:04.997+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:04.998+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:05.657+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:05.657+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:05.657+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:05.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:05.658+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:05.646+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:05.646+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:05.646+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:05.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:05.648+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:06.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:05.968+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:06.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:05.968+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:06.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:05.968+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:06.453 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:05.969+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:06.612+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:06.612+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:06.612+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:06.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:06.613+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:06.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:06.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:06.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:06.662+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:07.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:07.170+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15778 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:07.298 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:06.948+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:07.298 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:06.948+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:07.298 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:06.948+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:07.298 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:06.949+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:07.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:07.594+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:07.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:07.594+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:07.943 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:07.594+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:07.944 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:07.595+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:07.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:07.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:07.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:07.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:07.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:07.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:07.944 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:07.618+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:08.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:07.940+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:08.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:07.940+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:08.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:07.940+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:08.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:07.941+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:08.579+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:08.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:08.579+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:08.579+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:08.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:08.580+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:08.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:08.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:08.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:08.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:08.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:08.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:08.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:08.618+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:09.247 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:08.954+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:09.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:08.954+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:09.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:08.954+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:09.248 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:08.954+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:09.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:09.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:09.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:09.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:09.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:09.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:09.592+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:09.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:09.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:09.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:09.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:09.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:09.658+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:09.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:09.658+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:10.000+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:10.000+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:10.000+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:10.001+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:10.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:10.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:10.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:10.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:10.565+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:10.612+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:10.612+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:10.612+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:10.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:10.614+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:11.024+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:11.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:11.024+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:11.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:11.024+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:11.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:11.025+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:11.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:11.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:11.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:11.576+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:11.573+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:11.573+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:11.573+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:11.574+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:12.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:12.172+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15867 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:12.057+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:12.057+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:12.057+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:12.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:12.057+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:12.846 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:12.533+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:12.846 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:12.533+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:12.847 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:12.533+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:12.847 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:12.534+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:12.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:12.576+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:12.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:12.576+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:12.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:12.576+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:12.847 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:12.578+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:13.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:13.054+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:13.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:13.054+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:13.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:13.054+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:13.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:13.055+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:13.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:13.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:13.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:13.552+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:13.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:13.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:13.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:13.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:13.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:13.580+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:13.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:13.581+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:14.104+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:14.104+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:14.104+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:14.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:14.105+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:14.547+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:14.547+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:14.547+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:14.548+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:14.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:14.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:14.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:14.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:14.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:14.577+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:14.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:14.578+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:15.065+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:15.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:15.065+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:15.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:15.065+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:15.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:15.066+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:15.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:15.603+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:15.918 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:15.603+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:15.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:15.603+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:15.919 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:15.603+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:15.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:15.545+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:15.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:15.545+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:15.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:15.545+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:15.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:15.546+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:16.115+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:16.115+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:16.115+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:16.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:16.115+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:16.551+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:16.551+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:16.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:16.551+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:16.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:16.552+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:16.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:16.578+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:16.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:16.578+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:16.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:16.578+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:16.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:16.579+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:17.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:17.173+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 15958 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:17.140+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:17.140+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:17.140+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:17.140+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:17.545+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:17.545+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:17.545+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:17.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:17.546+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:17.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:17.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:17.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:17.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:17.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:17.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:17.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:17.528+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:18.162+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:18.162+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:18.162+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:18.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:18.163+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:18.533+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:18.533+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:18.533+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:18.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:18.534+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:18.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:18.529+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:18.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:18.529+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:18.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:18.529+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:18.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:18.530+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:19.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:19.171+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:19.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:19.171+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:19.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:19.171+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:19.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:19.172+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:19.530+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:19.530+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:19.530+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:19.531+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:19.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:19.564+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:19.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:19.564+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:19.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:19.564+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:19.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:19.566+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:20.160+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:20.160+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:20.160+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:20.160+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:20.495+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:20.495+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:20.495+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:20.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:20.496+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:20.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:20.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:20.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:20.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:20.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:20.593+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:20.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:20.594+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:21.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:21.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:21.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:21.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:21.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:21.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:21.546 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:21.199+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:21.543+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:21.543+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:21.543+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:21.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:21.544+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:21.634+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:21.634+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:21.634+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:21.635+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:22.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:22.175+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16047 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:22.249+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:22.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:22.249+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:22.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:22.249+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:22.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:22.250+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:22.502+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:22.502+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:22.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:22.502+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:22.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:22.503+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:22.608+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:22.608+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:22.608+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:22.608+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:23.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:23.238+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:23.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:23.238+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:23.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:23.238+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:23.595 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:23.239+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:23.595 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:23.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:23.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:23.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:23.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:23.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:23.596 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:23.470+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:23.627+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:23.627+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:23.627+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:23.628+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:24.601 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:24.458+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:24.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:24.458+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:24.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:24.458+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:24.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:24.458+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:24.602 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:24.238+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:24.602 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:24.238+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:24.602 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:24.238+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:24.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:24.239+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:24.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:24.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:24.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:24.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:24.600+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:25.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:25.273+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:25.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:25.273+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:25.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:25.273+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:25.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:25.273+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:25.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:25.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:25.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:25.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:25.601+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:25.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:25.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:25.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:25.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:25.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:25.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:25.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:25.442+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:26.585 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:26.265+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:26.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:26.265+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:26.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:26.265+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:26.586 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:26.265+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:26.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:26.485+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:26.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:26.485+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:26.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:26.485+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:26.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:26.486+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:26.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:26.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:26.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:26.583+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:26.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:26.583+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:27.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:27.177+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16135 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:27.310+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:27.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:27.310+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:27.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:27.310+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:27.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:27.311+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:27.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:27.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:27.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:27.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:27.474+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:27.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:27.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:27.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:27.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:27.586+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:28.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:28.297+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:28.553 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:28.297+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:28.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:28.297+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:28.554 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:28.297+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:28.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:28.438+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:28.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:28.438+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:28.554 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:28.438+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:28.555 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:28.439+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:28.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:28.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:28.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:28.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:28.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:28.551+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:28.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:28.552+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:29.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:29.396+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:29.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:29.396+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:29.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:29.396+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:29.571 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:29.397+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:29.571 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:29.343+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:29.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:29.343+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:29.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:29.343+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:29.572 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:29.343+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:29.568+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:29.568+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:29.568+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:29.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:29.569+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:30.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:30.334+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:30.426 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:30.334+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:30.427 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:30.334+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:30.427 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:30.334+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:30.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:30.424+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:30.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:30.424+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:30.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:30.424+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:30.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:30.425+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:30.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:30.598+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:30.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:30.598+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:30.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:30.598+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:30.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:30.599+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:31.607 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:31.475+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:31.607 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:31.475+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:31.607 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:31.475+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:31.607 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:31.476+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:31.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:31.331+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:31.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:31.331+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:31.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:31.331+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:31.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:31.331+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:31.604+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:31.604+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:31.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:31.604+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:31.605+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:32.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:32.178+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16223 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:32.368+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:32.368+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:32.368+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:32.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:32.368+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:32.462+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:32.462+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:32.462+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:32.463+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:32.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:32.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:32.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:32.571+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:33.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:33.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:33.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:33.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:33.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:33.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:33.612 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:33.479+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:33.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:33.357+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:33.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:33.357+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:33.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:33.357+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:33.613 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:33.357+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:33.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:33.609+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:33.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:33.609+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:33.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:33.609+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:33.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:33.611+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:34.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:34.360+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:34.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:34.360+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:34.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:34.360+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:34.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:34.361+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:34.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:34.512+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:34.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:34.512+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:34.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:34.512+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:34.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:34.513+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:34.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:34.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:34.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:34.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:34.646+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:35.684 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:35.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:35.684 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:35.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:35.684 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:35.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:35.684 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:35.470+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:35.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:35.382+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:35.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:35.382+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:35.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:35.382+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:35.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:35.383+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:35.681+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:35.681+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:35.681+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:35.682+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:36.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:36.354+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:36.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:36.354+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:36.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:36.354+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:36.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:36.355+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:36.655 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:36.423+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:36.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:36.423+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:36.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:36.423+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:36.656 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:36.424+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:36.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:36.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:36.652+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:36.653+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:37.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:37.180+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16314 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:37.313+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:37.313+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:37.313+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:37.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:37.314+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:37.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:37.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:37.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:37.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:37.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:37.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:37.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:37.406+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:37.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:37.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:37.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:37.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:37.674+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:38.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:38.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:38.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:38.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:38.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:38.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:38.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:38.442+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:38.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:38.327+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:38.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:38.327+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:38.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:38.327+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:38.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:38.328+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:38.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:38.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:38.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:38.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:38.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:38.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:38.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:38.626+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:39.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:39.378+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:39.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:39.378+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:39.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:39.378+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:39.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:39.378+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:39.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:39.453+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:39.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:39.453+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:39.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:39.453+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:39.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:39.454+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:39.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:39.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:39.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:39.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:39.629+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:40.419 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:40.362+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:40.419 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:40.362+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:40.419 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:40.362+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:40.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:40.362+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:40.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:40.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:40.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:40.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:40.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:40.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:40.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:40.418+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:40.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:40.611+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:40.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:40.611+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:40.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:40.611+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:40.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:40.612+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:41.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:41.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:41.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:41.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:41.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:41.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:41.657 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:41.425+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:41.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:41.403+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:41.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:41.403+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:41.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:41.403+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:41.658 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:41.404+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:41.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:41.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:41.654+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:41.655+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:42.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:42.182+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16402 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:42.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:42.397+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:42.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:42.397+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:42.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:42.397+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:42.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:42.398+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:42.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:42.394+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:42.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:42.394+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:42.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:42.394+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:42.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:42.395+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:42.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:42.610+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:42.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:42.610+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:42.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:42.610+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:42.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:42.611+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:43.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:43.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:43.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:43.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:43.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:43.602+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:43.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:43.364+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:43.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:43.364+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:43.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:43.364+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:43.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:43.364+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:43.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:43.440+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:43.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:43.440+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:43.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:43.440+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:43.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:43.441+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:44.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:44.579+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:44.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:44.579+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:44.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:44.579+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:44.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:44.580+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:44.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:44.390+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:44.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:44.390+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:44.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:44.390+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:44.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:44.391+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:44.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:44.395+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:44.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:44.395+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:44.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:44.395+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:44.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:44.395+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:45.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:45.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:45.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:45.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:45.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:45.580+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:45.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:45.581+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:45.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:45.347+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:45.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:45.347+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:45.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:45.347+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:45.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:45.348+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:45.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:45.439+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:45.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:45.439+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:45.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:45.439+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:45.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:45.440+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:46.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:46.466+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:46.599 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:46.466+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:46.599 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:46.466+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:46.599 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:46.466+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:46.599 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:46.332+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:46.599 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:46.332+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:46.599 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:46.332+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:46.600 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:46.333+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:46.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:46.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:46.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:46.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:46.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:46.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:46.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:46.597+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:47.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:47.184+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16492 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:47.287+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:47.287+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:47.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:47.287+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:47.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:47.287+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:47.581+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:47.581+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:47.581+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:47.581+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:47.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:47.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:47.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:47.473+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:48.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:48.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:48.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:48.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:48.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:48.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:48.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:48.427+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:48.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:48.298+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:48.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:48.298+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:48.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:48.298+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:48.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:48.299+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:48.623+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:48.623+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:48.623+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:48.624+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:49.601 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:49.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:49.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:49.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:49.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:49.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:49.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:49.417+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:49.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:49.293+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:49.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:49.293+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:49.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:49.293+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:49.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:49.294+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:49.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:49.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:49.599+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:49.600+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:50.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:50.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:50.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:50.245+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:50.452+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:50.452+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:50.452+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:50.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:50.452+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:50.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:50.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:50.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:50.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:50.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:50.629+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:50.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:50.629+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:51.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:51.272+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:51.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:51.272+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:51.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:51.272+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:51.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:51.273+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:51.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:51.455+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:51.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:51.455+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:51.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:51.455+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:51.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:51.456+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:51.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:51.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:51.624+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:51.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:51.625+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:52.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:52.185+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16581 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:52.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:52.291+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:52.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:52.291+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:52.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:52.291+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:52.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:52.292+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:52.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:52.428+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:52.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:52.428+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:52.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:52.428+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:52.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:52.429+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:52.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:52.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:52.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:52.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:52.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:52.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:52.632+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:53.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:53.307+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:53.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:53.307+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:53.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:53.307+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:53.598 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:53.307+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:53.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:53.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:53.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:53.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:53.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:53.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:53.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:53.459+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:53.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:53.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:53.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:53.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:53.602+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:54.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:54.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:54.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:54.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:54.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:54.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:54.588 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:54.445+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:54.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:54.284+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:54.284+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:54.284+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:54.589 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:54.285+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:54.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:54.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:54.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:54.585+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:54.587+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:55.623 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:55.443+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:55.623 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:55.443+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:55.623 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:55.443+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:55.623 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:55.444+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:55.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:55.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:55.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:55.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:55.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:55.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:55.624 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:55.267+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:55.620+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:55.620+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:55.620+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:55.621+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:56.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:56.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:56.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:56.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:56.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:56.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:56.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:56.442+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:56.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:56.282+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:56.631 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:56.282+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:56.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:56.282+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:56.632 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:56.282+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:56.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:56.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:56.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:56.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:56.629+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:57.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:06:57.187+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16672 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:06:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:57.321+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:57.321+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:57.321+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:57.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:57.321+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:57.470+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:57.470+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:57.470+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:57.471+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:57.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:57.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:57.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:57.647+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:58.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:58.433+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:58.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:58.433+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:58.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:58.433+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:58.649 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:58.435+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:58.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:58.320+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:58.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:58.320+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:58.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:58.320+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:58.650 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:58.321+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:58.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:58.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:58.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:58.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:58.647+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:06:59.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:59.325+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:59.605 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:59.325+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:59.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:59.325+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:06:59.609 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:06:59.326+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:59.609 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:59.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:59.609 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:59.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:59.609 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:59.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:06:59.609 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:06:59.475+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:06:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:59.602+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:59.602+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:59.602+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:06:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:06:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:06:59.603+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:00.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:00.317+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:00.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:00.317+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:00.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:00.317+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:00.432 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:00.318+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:00.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:00.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:00.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:00.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:00.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:00.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:00.701 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:00.591+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:00.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:00.429+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:00.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:00.429+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:00.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:00.429+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:00.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:00.430+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:01.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:01.310+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:01.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:01.310+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:01.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:01.310+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:01.604 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:01.311+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:01.604 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:01.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:01.604 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:01.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:01.604 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:01.444+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:01.605 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:01.445+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:01.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:01.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:01.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:01.602+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:02.268 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:02.189+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16760 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:02.597 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:02.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:02.597 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:02.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:02.597 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:02.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:02.597 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:02.451+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:02.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:02.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:02.598 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:02.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:02.598 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:02.266+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:02.598 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:02.266+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:02.594+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:02.594+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:02.594+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:02.595+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:03.597 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:03.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:03.598 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:03.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:03.598 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:03.236+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:03.598 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:03.237+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:03.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:03.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:03.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:03.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:03.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:03.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:03.598 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:03.417+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:03.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:03.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:03.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:03.642+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:04.646 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:04.429+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:04.646 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:04.429+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:04.646 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:04.429+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:04.646 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:04.429+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:04.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:04.219+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:04.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:04.219+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:04.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:04.219+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:04.647 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:04.219+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:04.643+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:04.643+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:04.643+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:04.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:04.644+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:05.226+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:05.226+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:05.226+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:05.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:05.227+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:05.457+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:05.457+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:05.457+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:05.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:05.458+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:05.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:05.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:05.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:05.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:05.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:05.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:05.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:05.694+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:06.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:06.412+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:06.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:06.412+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:06.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:06.412+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:06.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:06.413+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:06.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:06.242+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:06.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:06.242+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:06.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:06.242+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:06.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:06.243+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:07.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:06.719+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:07.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:06.719+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:07.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:06.719+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:07.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:06.720+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:07.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:07.191+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16851 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:07.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:07.366+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:07.366+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:07.366+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:07.367+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:07.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:07.223+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:07.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:07.223+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:07.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:07.223+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:07.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:07.223+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:08.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:07.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:08.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:07.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:08.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:07.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:08.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:07.735+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:08.408+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:08.408+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:08.408+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:08.409+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:08.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:08.191+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:08.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:08.191+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:08.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:08.191+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:08.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:08.192+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:09.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:08.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:09.148 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:08.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:09.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:08.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:09.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:08.764+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:09.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:09.369+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:09.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:09.369+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:09.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:09.369+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:09.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:09.370+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:09.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:09.146+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:09.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:09.146+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:09.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:09.146+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:09.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:09.146+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:10.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:09.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:10.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:09.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:10.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:09.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:10.165 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:09.800+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:10.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:10.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:10.405+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:10.405+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:10.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:10.179+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:10.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:10.179+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:10.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:10.179+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:10.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:10.180+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:11.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:10.793+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:11.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:10.793+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:11.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:10.793+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:11.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:10.794+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:11.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:11.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:11.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:11.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:11.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:11.425+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:11.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:11.425+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:11.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:11.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:11.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:11.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:11.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:11.208+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:11.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:11.208+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:11.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:11.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:11.798+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:11.799+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:12.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:12.193+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 16938 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:12.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:12.409+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:12.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:12.409+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:12.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:12.409+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:12.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:12.410+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:12.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:12.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:12.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:12.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:12.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:12.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:12.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:12.168+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:13.145 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:12.842+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:13.145 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:12.842+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:13.145 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:12.842+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:13.146 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:12.843+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:13.370+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:13.370+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:13.370+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:13.371+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:13.142+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:13.142+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:13.142+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:13.143+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:13.834+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:13.834+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:13.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:13.834+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:13.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:13.835+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:14.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:14.159+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:14.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:14.159+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:14.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:14.159+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:14.420 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:14.160+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:14.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:14.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:14.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:14.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:14.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:14.417+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:14.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:14.418+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:15.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:14.795+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:15.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:14.795+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:15.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:14.795+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:15.118 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:14.797+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:15.391 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:15.115+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:15.392 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:15.115+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:15.392 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:15.115+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:15.392 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:15.116+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:15.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:15.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:15.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:15.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:15.390+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:16.090 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:15.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:16.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:15.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:16.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:15.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:16.091 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:15.774+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:16.423 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:16.088+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:16.424 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:16.088+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:16.424 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:16.088+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:16.424 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:16.089+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:16.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:16.421+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:16.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:16.421+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:16.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:16.421+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:16.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:16.422+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:17.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:16.781+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:17.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:16.781+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:17.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:16.781+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:17.076 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:16.782+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:17.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:17.194+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17030 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:17.073+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:17.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:17.073+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:17.073+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:17.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:17.074+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:17.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:17.466+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:17.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:17.466+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:17.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:17.466+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:17.796 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:17.467+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:18.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:17.793+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:18.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:17.793+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:18.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:17.793+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:18.086 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:17.795+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:18.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:18.082+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:18.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:18.082+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:18.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:18.082+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:18.429 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:18.082+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:18.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:18.427+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:18.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:18.427+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:18.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:18.427+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:18.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:18.428+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:19.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:18.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:19.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:18.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:19.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:18.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:19.069 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:18.801+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:19.385 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:19.067+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:19.386 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:19.067+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:19.386 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:19.067+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:19.386 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:19.067+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:19.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:19.382+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:19.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:19.382+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:19.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:19.382+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:19.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:19.383+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:20.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:19.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:20.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:19.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:20.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:19.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:20.107 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:19.770+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:20.411 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:20.104+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:20.411 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:20.104+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:20.411 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:20.104+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:20.412 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:20.104+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:20.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:20.407+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:20.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:20.407+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:20.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:20.407+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:20.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:20.408+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:21.100 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:20.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:21.100 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:20.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:21.100 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:20.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:21.100 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:20.743+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:21.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:21.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:21.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:21.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:21.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:21.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:21.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:21.097+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:21.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:21.442+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:21.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:21.442+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:21.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:21.442+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:21.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:21.443+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:21.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:21.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:21.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:21.764+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:22.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:22.196+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17118 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:22.088+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:22.088+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:22.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:22.088+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:22.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:22.089+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:22.488+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:22.488+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:22.488+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:22.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:22.489+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:22.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:22.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:22.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:22.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:22.733+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:23.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:23.107+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:23.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:23.107+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:23.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:23.107+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:23.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:23.107+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:23.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:23.460+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:23.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:23.460+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:23.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:23.460+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:23.748 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:23.461+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:24.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:23.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:24.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:23.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:24.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:23.745+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:24.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:23.746+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:24.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:24.108+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:24.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:24.108+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:24.418 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:24.108+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:24.419 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:24.109+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:24.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:24.415+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:24.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:24.416+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:24.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:24.416+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:24.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:24.417+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:25.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:24.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:25.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:24.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:25.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:24.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:25.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:24.753+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:25.391 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:25.082+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:25.391 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:25.082+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:25.391 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:25.082+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:25.391 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:25.083+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:25.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:25.388+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:25.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:25.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:25.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:25.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:25.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:25.389+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:26.084 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:25.726+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:26.084 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:25.726+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:26.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:25.726+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:26.085 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:25.727+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:26.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:26.081+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:26.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:26.081+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:26.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:26.081+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:26.434 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:26.082+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:26.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:26.431+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:26.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:26.431+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:26.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:26.431+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:26.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:26.433+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:27.099 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:26.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:27.099 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:26.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:27.099 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:26.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:27.099 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:26.721+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:27.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:27.198+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17209 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:27.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:27.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:27.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:27.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:27.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:27.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:27.097+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:27.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:27.456+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:27.767 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:27.456+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:27.767 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:27.456+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:27.767 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:27.456+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:28.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:27.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:28.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:27.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:28.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:27.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:28.108 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:27.765+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:28.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:28.105+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:28.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:28.105+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:28.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:28.105+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:28.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:28.106+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:28.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:28.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:28.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:28.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:28.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:28.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:28.746 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:28.479+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:29.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:28.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:29.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:28.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:29.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:28.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:29.127 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:28.744+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:29.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:29.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:29.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:29.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:29.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:29.125+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:29.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:29.505+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:29.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:29.505+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:29.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:29.505+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:29.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:29.505+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:29.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:29.698+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:29.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:29.698+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:29.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:29.698+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:29.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:29.699+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:30.151+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:30.151+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:30.151+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:30.151+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:30.700+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:30.700+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:30.700+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:30.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:30.701+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:30.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:30.515+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:30.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:30.515+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:30.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:30.515+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:30.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:30.517+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:31.190+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:31.190+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:31.190+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:31.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:31.191+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:31.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:31.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:31.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:31.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:31.540+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:31.673+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:31.673+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:31.673+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:31.674+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:32.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:32.200+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17297 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:32.151+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:32.151+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:32.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:32.151+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:32.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:32.151+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:32.555+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:32.555+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:32.555+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:32.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:32.555+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:32.680+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:32.680+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:32.680+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:32.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:32.681+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:33.130+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:33.130+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:33.130+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:33.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:33.131+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:33.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:33.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:33.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:33.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:33.594+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:33.648+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:33.648+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:33.648+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:33.649+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:34.169+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:34.169+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:34.169+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:34.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:34.170+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:34.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:34.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:34.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:34.642+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:34.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:34.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:34.674+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:34.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:34.675+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:35.180+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:35.180+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:35.180+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:35.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:35.180+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:35.655+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:35.655+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:35.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:35.655+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:35.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:35.656+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:35.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:35.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:35.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:35.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:35.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:35.688+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:35.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:35.690+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:36.135+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:36.135+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:36.135+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:36.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:36.136+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:36.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:36.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:36.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:36.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:36.627+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:36.660+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:36.660+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:36.660+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:36.661+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:37.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:37.202+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17388 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:37.099+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:37.099+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:37.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:37.099+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:37.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:37.099+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:37.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:37.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:37.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:37.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:37.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:37.619+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:37.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:37.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:37.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:37.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:37.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:37.644+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:37.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:37.646+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:38.072+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:38.072+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:38.072+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:38.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:38.073+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:38.663+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:38.663+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:38.663+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:38.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:38.664+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:38.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:38.683+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:38.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:38.683+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:38.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:38.683+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:38.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:38.684+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:39.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:39.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:39.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:39.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:39.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:39.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:39.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:39.096+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:39.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:39.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:39.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:39.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:39.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:39.667+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:39.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:39.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:39.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:39.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:39.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:39.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:39.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:39.733+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:40.089+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:40.089+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:40.089+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:40.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:40.090+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:40.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:40.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:40.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:40.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:40.687+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:40.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:40.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:40.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:40.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:40.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:40.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:40.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:40.751+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:41.068+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:41.068+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:41.068+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:41.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:41.069+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:41.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:41.648+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:41.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:41.648+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:41.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:41.648+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:41.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:41.649+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:41.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:41.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:41.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:41.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:41.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:41.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:41.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:41.765+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:42.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:42.114+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:42.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:42.114+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:42.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:42.114+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:42.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:42.114+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:42.610 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:42.204+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17476 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:42.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:42.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:42.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:42.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:42.608+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:42.737+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:42.737+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:42.737+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:42.738+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:43.114+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:43.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:43.114+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:43.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:43.114+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:43.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:43.114+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:43.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:43.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:43.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:43.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:43.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:43.733+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:43.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:43.634+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:43.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:43.634+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:43.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:43.634+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:43.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:43.635+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:44.091+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:44.091+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:44.091+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:44.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:44.092+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:44.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:44.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:44.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:44.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:44.755+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:44.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:44.607+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:44.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:44.607+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:44.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:44.607+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:44.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:44.608+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:45.057+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:45.057+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:45.057+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:45.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:45.057+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:45.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:45.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:45.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:45.618+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:45.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:45.801+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:45.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:45.801+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:45.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:45.801+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:45.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:45.802+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:46.025+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:46.025+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:46.025+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:46.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:46.025+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:46.623+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:46.623+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:46.623+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:46.624+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:46.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:46.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:46.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:46.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:46.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:46.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:46.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:46.784+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:47.335 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:47.206+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17566 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:47.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:47.011+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:47.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:47.011+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:47.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:47.011+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:47.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:47.011+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:47.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:47.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:47.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:47.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:47.786+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:47.632+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:47.632+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:47.632+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:47.632+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:48.020+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:48.020+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:48.020+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:48.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:48.020+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:48.671+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:48.671+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:48.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:48.671+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:48.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:48.672+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:48.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:48.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:48.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:48.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:48.740+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:49.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:48.995+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:49.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:48.995+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:49.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:48.995+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:49.249 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:48.996+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:49.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:49.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:49.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:49.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:49.741+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:49.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:49.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:49.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:49.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:49.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:49.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:49.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:49.661+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:50.023+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:50.023+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:50.023+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:50.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:50.024+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:50.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:50.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:50.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:50.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:50.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:50.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:50.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:50.776+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:50.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:50.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:50.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:50.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:50.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:50.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:50.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:50.667+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:51.031+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:51.031+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:51.031+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:51.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:51.032+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:51.620+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:51.620+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:51.620+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:51.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:51.620+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:51.825+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:51.825+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:51.825+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:51.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:51.826+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:52.335 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:52.208+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17654 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:52.335 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:52.073+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:52.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:52.073+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:52.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:52.073+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:52.336 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:52.074+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:52.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:52.615+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:52.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:52.615+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:52.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:52.615+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:52.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:52.616+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:53.123 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:52.868+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:53.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:52.868+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:53.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:52.868+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:53.124 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:52.869+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:53.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:53.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:53.121+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:53.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:53.122+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:53.645+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:53.645+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:53.645+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:53.646+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:53.845+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:53.845+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:53.845+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:53.845+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:54.158+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:54.158+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:54.158+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:54.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:54.159+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:54.690+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:54.690+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:54.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:54.690+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:54.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:54.691+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:54.847+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:54.847+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:54.847+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:54.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:54.847+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:55.137+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:55.137+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:55.137+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:55.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:55.138+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:55.662+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:55.662+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:55.662+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:55.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:55.663+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:55.818+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:55.818+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:55.818+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:55.819+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:56.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:56.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:56.167+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:56.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:56.167+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:56.657+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:56.657+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:56.657+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:56.658+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:56.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:56.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:56.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:56.863+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:57.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:07:57.210+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17746 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:07:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:57.132+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:57.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:57.132+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:57.132+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:57.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:57.132+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:58.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:57.814+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:58.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:57.814+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:58.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:57.814+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:58.178 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:57.815+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:58.178 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:57.705+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:58.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:57.705+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:58.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:57.705+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:58.179 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:57.706+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:58.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:58.175+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:58.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:58.175+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:58.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:58.175+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:58.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:58.176+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:59.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:58.740+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:59.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:58.740+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:59.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:58.740+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:07:59.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:58.741+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:07:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:58.813+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:58.813+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:58.813+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:07:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:58.814+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:07:59.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:59.197+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:59.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:59.197+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:59.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:59.197+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:07:59.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:07:59.197+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:00.038 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:59.835+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:00.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:59.835+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:00.040 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:59.835+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:00.041 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:07:59.836+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:00.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:59.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:00.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:59.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:00.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:59.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:00.041 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:07:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:07:59.708+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:00.366 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:00.209+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:00.366 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:00.209+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:00.367 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:00.209+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:00.367 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:00.209+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:00.722+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:00.722+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:00.722+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:01.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:00.724+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:01.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:00.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:01.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:00.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:01.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:00.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:01.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:00.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:01.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:01.213+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:01.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:01.213+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:01.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:01.213+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:01.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:01.214+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:01.691+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:01.691+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:01.691+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:01.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:01.692+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:01.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:01.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:01.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:01.786+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:02.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:02.212+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17833 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:02.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:02.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:02.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:02.243+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:02.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:02.244+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:02.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:02.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:02.729+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:02.729+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:03.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:02.758+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:03.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:02.758+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:03.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:02.758+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:03.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:02.759+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:03.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:03.278+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:03.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:03.278+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:03.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:03.278+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:03.603 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:03.280+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:03.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:03.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:03.707+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:03.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:03.708+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:03.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:03.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:03.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:03.733+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:04.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:04.303+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:04.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:04.303+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:04.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:04.303+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:04.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:04.304+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:04.746+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:04.746+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:04.746+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:05.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:04.747+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:05.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:04.714+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:05.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:04.714+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:05.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:04.714+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:05.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:04.716+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:05.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:05.325+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:05.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:05.325+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:05.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:05.325+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:05.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:05.326+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:05.765+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:05.765+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:06.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:05.765+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:06.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:05.766+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:05.717+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:05.717+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:05.717+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:05.717+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:06.350+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:06.350+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:06.350+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:06.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:06.350+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:07.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:06.741+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:07.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:06.741+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:07.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:06.741+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:07.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:06.742+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:07.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:06.735+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:07.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:06.736+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:07.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:06.736+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:07.205 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:06.737+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:07.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:07.214+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 17923 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:07.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:07.333+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:07.333+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:07.333+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:07.333+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:07.777+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:07.777+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:07.777+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:08.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:07.778+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:08.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:07.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:08.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:07.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:08.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:07.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:08.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:07.728+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:08.333+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:08.334+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:08.334+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:08.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:08.334+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:08.820+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:08.820+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:08.820+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:09.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:08.821+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:08.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:08.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:08.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:08.753+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:09.368+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:09.368+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:09.368+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:09.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:09.369+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:10.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:09.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:10.169 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:09.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:10.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:09.802+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:10.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:09.803+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:10.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:09.704+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:10.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:09.704+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:10.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:09.704+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:10.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:09.705+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:10.370+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:10.370+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:10.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:10.370+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:10.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:10.371+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:10.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:10.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:10.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:10.779+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:11.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:10.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:11.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:10.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:11.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:10.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:11.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:10.741+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:11.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:11.409+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:11.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:11.409+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:11.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:11.409+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:11.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:11.410+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:11.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:11.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:11.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:11.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:11.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:11.815+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:11.756+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:11.756+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:11.756+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:11.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:11.756+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:12.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:12.368+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:12.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:12.368+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:12.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:12.368+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:12.655 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:12.369+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:12.655 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:12.216+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18010 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:12.795+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:12.795+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:12.795+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:12.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:12.797+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:12.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:12.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:12.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:12.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:12.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:12.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:12.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:12.735+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:13.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:13.394+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:13.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:13.394+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:13.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:13.394+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:13.395+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:14.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:13.715+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:14.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:13.715+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:14.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:13.715+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:14.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:13.717+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:13.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:13.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:13.754+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:13.755+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:14.442+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:14.442+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:14.442+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:14.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:14.443+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:14.739+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:15.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:14.739+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:14.739+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:14.741+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:15.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:14.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:15.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:14.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:15.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:14.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:15.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:14.721+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:15.430+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:15.430+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:15.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:15.430+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:15.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:15.431+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:15.744+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:16.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:15.744+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:16.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:15.744+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:16.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:15.745+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:16.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:15.712+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:16.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:15.712+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:16.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:15.712+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:16.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:15.712+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:16.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:16.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:16.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:16.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:16.765 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:16.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:16.766 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:16.687+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:16.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:16.473+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:16.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:16.473+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:16.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:16.473+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:16.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:16.473+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:17.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:16.762+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:17.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:16.762+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:17.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:16.762+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:17.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:16.763+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:17.688 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:17.218+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18101 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:17.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:17.428+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:17.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:17.428+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:17.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:17.428+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:17.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:17.429+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:17.743+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:17.743+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:17.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:17.743+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:17.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:17.744+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:17.685+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:17.685+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:17.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:17.685+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:17.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:17.686+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:18.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:18.706+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:18.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:18.706+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:18.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:18.706+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:18.777 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:18.707+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:18.778 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:18.473+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:18.778 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:18.473+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:18.778 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:18.473+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:18.778 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:18.473+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:18.774+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:18.774+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:19.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:18.774+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:19.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:18.775+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:19.440+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:19.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:19.440+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:19.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:19.440+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:19.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:19.440+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:20.169 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:19.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:20.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:19.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:20.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:19.732+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:20.170 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:19.733+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:20.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:19.821+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:20.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:19.821+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:20.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:19.821+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:20.171 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:19.823+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:20.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:20.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:20.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:20.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:20.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:20.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:20.749 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:20.459+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:20.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:21.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:20.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:21.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:20.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:21.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:20.747+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:21.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:20.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:21.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:20.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:21.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:20.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:21.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:20.860+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:21.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:21.446+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:21.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:21.446+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:21.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:21.446+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:21.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:21.446+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:21.829+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:21.829+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:21.829+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:21.831+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:21.749+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:21.749+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:21.749+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:21.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:21.750+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:22.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:22.219+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18188 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:22.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:22.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:22.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:22.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:22.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:22.462+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:22.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:22.463+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:23.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:22.864+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:23.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:22.864+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:23.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:22.864+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:23.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:22.865+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:22.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:22.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:22.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:22.702+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:23.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:23.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:23.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:23.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:23.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:23.500+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:23.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:23.501+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:23.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:23.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:23.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:23.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:23.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:23.720+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:23.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:23.721+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:24.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:23.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:24.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:23.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:24.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:23.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:24.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:23.908+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:24.745 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:24.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:24.745 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:24.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:24.746 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:24.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:24.746 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:24.484+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:25.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:24.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:25.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:24.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:24.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:24.744+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:24.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:24.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:24.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:24.904+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:25.759 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:25.459+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:25.759 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:25.459+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:25.759 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:25.459+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:25.760 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:25.460+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:25.892+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:25.892+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:25.892+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:26.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:25.893+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:26.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:25.757+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:26.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:25.757+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:26.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:25.757+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:26.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:25.757+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:26.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:26.498+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:26.762 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:26.498+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:26.763 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:26.498+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:26.763 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:26.498+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:27.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:26.920+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:27.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:26.920+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:27.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:26.920+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:27.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:26.921+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:27.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:26.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:27.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:26.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:27.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:26.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:27.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:26.761+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:27.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:27.222+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18279 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:27.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:27.543+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:27.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:27.543+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:27.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:27.543+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:27.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:27.544+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:27.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:27.722+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:27.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:27.722+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:27.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:27.722+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:27.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:27.723+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:27.913+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:27.913+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:27.913+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:28.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:27.913+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:28.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:28.739+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:28.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:28.739+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:28.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:28.739+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:28.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:28.740+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:28.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:28.527+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:28.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:28.527+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:28.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:28.527+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:28.875 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:28.528+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:28.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:28.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:28.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:29.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:28.873+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:29.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:29.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:29.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:29.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:29.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:29.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:29.900 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:29.702+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:29.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:29.577+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:29.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:29.577+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:29.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:29.577+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:29.901 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:29.578+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:30.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:29.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:30.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:29.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:30.171 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:29.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:30.171 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:29.899+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:30.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:30.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:30.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:30.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:30.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:30.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:30.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:30.547+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:30.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:30.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:30.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:30.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:30.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:30.727+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:30.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:30.728+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:30.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:30.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:30.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:31.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:30.925+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:31.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:31.514+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:31.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:31.514+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:31.912 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:31.514+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:31.912 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:31.515+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:31.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:31.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:31.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:31.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:31.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:31.760+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:31.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:31.761+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:32.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:31.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:32.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:31.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:32.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:31.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:32.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:31.910+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:32.535 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:32.224+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18367 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:32.870 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:32.739+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:32.871 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:32.739+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:32.871 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:32.739+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:32.871 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:32.740+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:32.871 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:32.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:32.871 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:32.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:32.871 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:32.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:32.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:32.533+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:32.868+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:32.868+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:32.868+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:33.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:32.869+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:33.788 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:33.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:33.788 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:33.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:33.788 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:33.487+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:33.788 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:33.488+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:34.159 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:33.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:34.160 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:33.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:34.160 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:33.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:34.160 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:33.908+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:34.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:33.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:34.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:33.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:34.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:33.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:34.161 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:33.786+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:34.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:34.516+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:34.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:34.516+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:34.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:34.516+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:34.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:34.516+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:34.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:34.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:34.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:34.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:34.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:34.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:34.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:34.738+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:34.942+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:34.942+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:34.942+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:34.943+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:35.786 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:35.524+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:35.787 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:35.524+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:35.787 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:35.524+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:35.787 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:35.525+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:35.912+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:35.912+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:35.912+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:36.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:35.913+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:36.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:35.784+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:36.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:35.784+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:36.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:35.784+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:36.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:35.785+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:36.748+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:36.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:36.748+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:36.748+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:36.749+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:36.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:36.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:36.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:36.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:36.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:36.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:36.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:36.534+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:37.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:36.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:37.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:36.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:37.228 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:36.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:37.229 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:36.964+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:37.524 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:37.226+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18459 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:37.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:37.521+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:37.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:37.521+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:37.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:37.521+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:37.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:37.522+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:37.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:37.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:37.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:37.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:37.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:37.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:37.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:37.709+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:38.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:37.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:38.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:37.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:38.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:37.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:38.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:37.982+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:38.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:38.676+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:38.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:38.676+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:38.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:38.676+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:38.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:38.677+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:38.942 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:38.560+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:38.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:38.560+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:38.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:38.560+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:38.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:38.560+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:38.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:38.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:38.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:38.940+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:39.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:39.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:39.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:39.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:39.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:39.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:39.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:39.647+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:39.936 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:39.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:39.936 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:39.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:39.936 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:39.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:39.937 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:39.548+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:40.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:39.934+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:40.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:39.934+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:40.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:39.934+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:40.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:39.934+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:40.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:40.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:40.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:40.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:40.894 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:40.631+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:40.894 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:40.632+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:40.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:40.596+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:40.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:40.596+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:40.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:40.596+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:40.894 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:40.597+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:40.890+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:40.890+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:40.890+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:40.890+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:41.910 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:41.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:41.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:41.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:41.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:41.625+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:41.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:41.625+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:41.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:41.627+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:41.911 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:41.627+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:41.912 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:41.627+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:41.912 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:41.628+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:42.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:41.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:42.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:41.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:42.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:41.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:42.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:41.908+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:42.584 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:42.227+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18547 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:42.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:42.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:42.911 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:42.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:42.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:42.661+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:42.912 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:42.661+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:42.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:42.580+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:42.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:42.580+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:42.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:42.580+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:42.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:42.581+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:42.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:42.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:42.908+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:42.908+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:43.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:43.617+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:43.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:43.617+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:43.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:43.617+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:43.692 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:43.618+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:43.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:43.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:43.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:43.902+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:43.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:43.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:43.689+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:43.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:43.689+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:44.866 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:44.725+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:44.866 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:44.725+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:44.866 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:44.725+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:44.866 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:44.726+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:44.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:44.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:44.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:44.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:44.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:44.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:44.867 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:44.588+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:44.863+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:44.863+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:44.863+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:45.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:44.864+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:45.840+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:45.840+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:45.840+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:45.841+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:45.619+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:45.619+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:45.619+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:45.619+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:45.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:45.728+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:45.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:45.728+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:45.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:45.728+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:45.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:45.729+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:46.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:46.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:46.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:46.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:46.665+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:47.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:46.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:47.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:46.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:47.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:46.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:47.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:46.890+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:47.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:46.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:47.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:46.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:47.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:46.773+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:47.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:46.774+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:47.623 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:47.229+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18638 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:47.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:47.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:47.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:47.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:47.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:47.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:47.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:47.621+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:47.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:47.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:47.845+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:47.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:47.846+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:47.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:47.791+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:47.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:47.791+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:47.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:47.791+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:47.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:47.792+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:48.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:48.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:48.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:48.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:48.812+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:48.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:48.846+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:48.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:48.846+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:48.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:48.846+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:48.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:48.848+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:48.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:48.644+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:48.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:48.644+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:48.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:48.644+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:48.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:48.644+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:49.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:49.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:49.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:49.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:49.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:49.854+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:49.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:49.854+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:49.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:49.640+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:49.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:49.640+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:49.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:49.640+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:49.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:49.641+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:49.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:49.797+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:49.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:49.797+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:49.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:49.797+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:49.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:49.797+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:50.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:50.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:50.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:50.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:50.903+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:50.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:50.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:50.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:50.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:50.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:50.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:50.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:50.684+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:50.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:50.772+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:50.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:50.772+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:50.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:50.772+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:50.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:50.773+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:52.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:51.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:52.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:51.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:52.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:51.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:52.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:51.873+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:52.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:51.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:52.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:51.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:52.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:51.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:52.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:51.729+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:52.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:51.753+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:52.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:51.753+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:52.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:51.753+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:52.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:51.754+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:52.590 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:52.231+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18726 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:52.850 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:52.721+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:52.850 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:52.721+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:52.851 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:52.721+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:52.851 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:52.722+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:52.851 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:52.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:52.851 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:52.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:52.851 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:52.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:52.851 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:52.729+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:52.848+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:52.848+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:52.848+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:53.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:52.849+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:53.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:53.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:53.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:53.831+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:53.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:53.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:53.743+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:53.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:53.744+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:53.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:53.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:53.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:53.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:53.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:53.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:21.009446+0000 (oldest deadline 2023-12-19T18:58:41.009446+0000) 2023-12-19T19:08:53.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:53.705+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:54.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:54.668+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:54.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:54.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:54.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:54.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:54.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:54.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:54.953 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:54.833+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:54.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:54.777+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:54.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:54.777+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:54.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:54.777+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:54.954 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:54.778+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:55.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:55.632+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:55.767+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:55.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:55.767+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:55.767+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:55.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:55.768+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:55.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:55.807+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:55.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:55.807+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:55.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:55.807+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:55.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:55.808+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:56.677+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:56.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:56.806+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:56.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:56.806+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:56.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:56.806+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:56.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:56.807+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:56.767+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:56.767+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:56.767+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:56.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:56.768+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:57.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:08:57.233+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18818 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:08:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:57.646+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:57.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:57.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:57.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:57.774+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:57.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:57.775+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:57.850+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:57.850+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:57.850+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:57.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:57.851+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:58.597+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:58.819+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:58.819+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:58.819+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:58.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:58.820+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:58.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:58.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:58.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:58.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:58.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:58.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:58.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:58.783+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:08:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:08:59.601+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:08:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:59.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:59.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:59.752+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:08:59.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:08:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:08:59.753+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:08:59.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:59.771+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:59.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:59.771+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:59.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:59.771+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:08:59.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:08:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:08:59.771+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:00.569+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:00.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:00.773+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:00.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:00.773+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:00.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:00.773+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:00.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:00.774+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:00.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:00.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:00.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:00.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:00.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:00.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:00.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:00.747+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:01.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:01.769+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:01.769+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:01.769+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:01.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:01.769+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:01.614+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:01.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:01.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:01.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:01.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:01.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:01.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:01.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:01.732+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:02.619 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:02.235+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18906 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:02.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:02.617+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:02.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:02.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:02.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:02.782+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:02.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:02.783+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:02.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:02.733+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:02.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:02.733+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:02.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:02.733+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:02.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:02.734+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:03.622+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:03.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:03.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:03.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:03.771+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:03.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:03.771+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:03.740+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:03.740+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:03.740+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:03.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:03.741+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:04.878 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:04.592+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:04.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:04.819+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:04.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:04.819+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:04.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:04.819+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:04.879 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:04.820+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:04.879 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:04.751+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:04.880 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:04.751+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:04.880 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:04.751+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:04.880 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:04.752+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:05.868 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:05.615+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:05.868 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:05.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:05.868 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:05.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:05.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:05.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:05.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:05.711+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:06.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:05.865+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:06.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:05.865+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:06.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:05.865+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:06.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:05.866+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:06.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:06.642+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:06.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:06.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:06.757+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:06.758+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:06.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:06.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:06.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:06.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:06.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:06.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.107030+0000 (oldest deadline 2023-12-19T18:58:54.107030+0000) 2023-12-19T19:09:06.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:06.863+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:07.613 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:07.237+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 18997 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:07.820+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:07.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:07.611+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:07.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:07.783+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:07.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:07.783+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:07.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:07.783+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T18:58:34.644562+0000 (oldest deadline 2023-12-19T18:58:54.644562+0000) 2023-12-19T19:09:07.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:07.784+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:08.845 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:08.770+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:08.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:08.573+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:09.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:08.842+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:09.800 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:09.788+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:09.801 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:09.540+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:10.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:09.798+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:10.842 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:10.588+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:10.842 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:10.783+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:11.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:10.840+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:11.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:11.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:11.734+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:11.561+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:12.594 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:12.239+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19085 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:12.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:12.841+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:12.755+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:12.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:12.591+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:13.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:13.817+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:13.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:13.768+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:13.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:13.608+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:14.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:14.818+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:14.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:14.860+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:14.618+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:15.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:15.894+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:15.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:15.659+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:15.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:15.786+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:16.665+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:16.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:16.779+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:16.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:16.875+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:17.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:17.242+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19176 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:17.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:17.813+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:17.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:17.837+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:17.637+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:18.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:18.851+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:18.684+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:18.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:18.811+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:19.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:19.818+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:19.679+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:19.765+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:20.764+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:20.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:20.834+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:20.630+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:20.630+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:20.631+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:20.631+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:21.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:21.788+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:21.792+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:21.671+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:21.671+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:21.671+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:21.672+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:22.694 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:22.244+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19263 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:22.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:22.810+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:22.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:22.816+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:22.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:22.692+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:22.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:22.692+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:22.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:22.692+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:22.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:22.692+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:23.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:23.800+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:23.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:23.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:23.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:23.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:23.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:23.697+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:24.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:24.813+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:24.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:24.765+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:24.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:24.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:24.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:24.674+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:25.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:25.852+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:25.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:25.799+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:25.647+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:25.647+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:25.647+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:25.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:25.648+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:26.860+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:26.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:26.754+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:26.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:26.651+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:26.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:26.651+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:26.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:26.651+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:26.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:26.652+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:27.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:27.246+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19353 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:27.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:27.752+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:27.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:27.879+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:27.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:27.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:27.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:27.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:27.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:27.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:27.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:27.695+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:28.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:28.830+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:28.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:28.743+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:28.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:28.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:28.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:28.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:28.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:28.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:28.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:28.668+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:29.731+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:29.731+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:29.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:29.731+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:29.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:29.732+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:29.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:29.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:29.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:29.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:29.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:29.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:29.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:29.669+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:29.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:29.824+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:29.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:29.824+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:29.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:29.824+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:29.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:29.826+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:30.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:30.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:30.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:30.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:30.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:30.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:30.697+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:30.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:30.861+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:30.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:30.861+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:30.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:30.861+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:30.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:30.863+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:30.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:30.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:30.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:30.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:30.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:30.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:30.953 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:30.761+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:31.869+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:31.869+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:31.869+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:31.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:31.870+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:31.726+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:31.726+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:31.726+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:31.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:31.727+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:31.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:31.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:31.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:31.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:31.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:31.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:31.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:31.697+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:32.527 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:32.248+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19440 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:32.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:32.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:32.904 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:32.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:32.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:32.761+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:32.905 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:32.762+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:32.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:32.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:32.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:32.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:32.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:32.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:32.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:32.716+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:32.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:32.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:33.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:32.901+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:33.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:32.903+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:33.939 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:33.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:33.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:33.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:33.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:33.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:33.940 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:33.813+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:33.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:33.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:33.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:33.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:33.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:33.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:33.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:33.757+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:33.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:33.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:33.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:34.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:33.938+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:34.942+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:34.942+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:34.942+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:34.943+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:35.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:34.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:35.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:34.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:35.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:34.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:35.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:34.740+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:35.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:34.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:35.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:34.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:35.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:34.855+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:35.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:34.855+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:35.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:35.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:35.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:35.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:35.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:35.823+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:35.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:35.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:35.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:35.783+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:35.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:35.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:35.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:36.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:35.969+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:36.985+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:36.985+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:36.985+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:37.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:36.985+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:36.822+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:36.822+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:36.822+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:36.823+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:37.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:36.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:37.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:36.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:37.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:36.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:37.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:36.778+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:37.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:37.250+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19531 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:37.948+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:37.948+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:37.948+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:37.949+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:38.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:37.869+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:38.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:37.869+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:38.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:37.869+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:38.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:37.869+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:38.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:37.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:38.204 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:37.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:38.204 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:37.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:38.204 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:37.738+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:38.954+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:38.954+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:38.954+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:38.955+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:39.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:38.765+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:39.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:38.765+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:39.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:38.765+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:39.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:38.765+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:39.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:38.888+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:39.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:38.888+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:39.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:38.888+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:39.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:38.889+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:40.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:39.916+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:40.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:39.916+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:40.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:39.916+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:40.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:39.917+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:40.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:39.927+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:40.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:39.927+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:40.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:39.927+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:40.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:39.928+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:40.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:39.810+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:40.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:39.810+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:40.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:39.810+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:40.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:39.810+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:40.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:40.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:41.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:40.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:41.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:40.966+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:41.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:40.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:41.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:40.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:41.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:40.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:41.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:40.905+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:41.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:40.843+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:41.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:40.843+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:41.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:40.843+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:41.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:40.843+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:41.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:41.935+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:41.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:41.935+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:41.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:41.935+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:41.968 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:41.935+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:41.968 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:41.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:41.968 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:41.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:41.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:41.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:41.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:41.797+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:42.256 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:41.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:42.256 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:41.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:42.256 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:41.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:42.256 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:41.965+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:42.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:42.252+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19618 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:43.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:42.990+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:43.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:42.990+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:43.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:42.990+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:43.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:42.991+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:43.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:42.929+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:43.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:42.929+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:43.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:42.929+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:43.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:42.931+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:43.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:42.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:43.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:42.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:43.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:42.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:43.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:42.843+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:44.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:43.873+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:44.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:43.873+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:44.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:43.873+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:44.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:43.873+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:44.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:43.961+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:44.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:43.961+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:44.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:43.961+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:44.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:43.962+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:44.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:43.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:44.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:43.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:44.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:43.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:44.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:43.957+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:44.919+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:44.919+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:44.919+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:45.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:44.920+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:44.971+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:44.971+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:44.971+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:44.973+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:45.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:44.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:45.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:44.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:45.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:44.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:45.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:44.863+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:46.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:45.858+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:46.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:45.858+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:46.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:45.858+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:46.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:45.859+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:45.996+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:45.996+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:45.996+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:45.997+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:46.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:45.899+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:46.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:45.899+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:46.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:45.899+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:46.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:45.899+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:46.902+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:46.902+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:46.902+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:47.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:46.903+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:47.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:46.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:47.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:46.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:47.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:46.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:47.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:46.833+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:47.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:47.005+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:47.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:47.005+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:47.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:47.005+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:47.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:47.006+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:47.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:47.254+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19708 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:48.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:47.847+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:48.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:47.847+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:48.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:47.847+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:48.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:47.847+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:47.875+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:47.875+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:47.875+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:48.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:47.875+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:48.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:48.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:48.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:48.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:48.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:48.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:48.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:48.026+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:48.908+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:48.908+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:48.908+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:49.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:48.909+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:48.884+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:48.884+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:48.884+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:49.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:48.885+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:49.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:48.994+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:49.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:48.994+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:49.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:48.994+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:49.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:48.995+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:50.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:49.926+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:50.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:49.926+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:50.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:49.926+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:50.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:49.927+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:50.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:49.993+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:50.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:49.993+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:50.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:49.993+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:50.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:49.994+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:49.953+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:49.953+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:49.953+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:50.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:49.954+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:51.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:51.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:51.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:51.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:51.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:51.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:51.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:51.044+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:51.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:50.903+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:51.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:50.903+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:51.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:50.903+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:51.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:50.903+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:51.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:50.994+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:51.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:50.994+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:51.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:50.994+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:51.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:50.995+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:51.959 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:51.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:51.959 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:51.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:51.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:51.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:51.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:51.907+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:52.259 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:52.056+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:52.259 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:52.056+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:52.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:52.056+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:52.260 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:52.056+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:52.260 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:51.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:52.260 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:51.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:52.260 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:51.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:52.260 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:51.957+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:52.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:52.256+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19795 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:53.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:52.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:53.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:52.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:53.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:52.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:53.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:52.986+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:53.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:53.073+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:53.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:53.073+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:53.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:53.073+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:53.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:53.074+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:53.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:52.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:53.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:52.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:53.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:52.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:53.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:52.890+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:53.954+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:53.954+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:53.954+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:53.955+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:54.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:53.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:54.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:53.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:54.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:53.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:54.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:53.903+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:54.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:54.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:54.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:54.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:54.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:54.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:54.053+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:55.049+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:55.049+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:55.049+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:55.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:55.049+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:55.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:54.955+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:55.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:54.955+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:55.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:54.955+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:55.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:54.956+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:55.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:54.861+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:55.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:54.861+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:55.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:54.861+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:55.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:54.861+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:55.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:55.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:56.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:55.864+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:56.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:55.865+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:56.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:56.090+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:56.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:56.090+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:56.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:56.090+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:56.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:56.091+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:56.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:55.999+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:56.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:55.999+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:56.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:55.999+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:56.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:56.000+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:57.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:57.093+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:57.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:57.093+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:57.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:57.093+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:57.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:57.094+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:57.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:56.959+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:57.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:56.959+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:57.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:56.959+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:57.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:56.960+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:57.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:56.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:57.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:56.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:57.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:56.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:57.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:56.876+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:57.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:09:57.259+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19888 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:09:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:57.925+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:57.925+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:57.925+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:57.927+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:58.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:57.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:58.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:57.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:58.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:57.842+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:58.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:57.843+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:58.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:58.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:58.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:58.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:58.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:58.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:58.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:58.081+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:59.121 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:58.969+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:59.122 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:58.969+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:59.122 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:58.969+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:09:59.122 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:58.970+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:09:59.122 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:58.855+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:59.122 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:58.855+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:59.122 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:58.855+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:09:59.122 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:58.856+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:09:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:59.119+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:59.119+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:59.119+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:09:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:09:59.120+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:00.102 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:00.099+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:00.102 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:00.099+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:00.102 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:00.099+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:00.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:59.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:00.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:59.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:00.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:59.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:00.103 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:09:59.956+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:00.103 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:59.825+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:00.103 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:59.825+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:00.104 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:59.825+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:00.104 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:09:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:09:59.825+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:00.101+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:01.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:00.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:01.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:00.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:01.119 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:00.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:01.120 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:00.861+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:01.120 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:01.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:01.120 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:01.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:01.120 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:01.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:01.120 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:01.004+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:01.115+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:01.115+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:01.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:01.117+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:02.004 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:01.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:02.004 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:01.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:02.004 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:01.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:02.004 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:01.860+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:02.263 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:02.001+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:02.264 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:02.001+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:02.264 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:02.001+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:02.264 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:02.002+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:02.264 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:02.130+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:02.264 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:02.130+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:02.265 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:02.130+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:02.265 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:02.131+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:02.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:02.261+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 19977 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:03.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:03.092+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:03.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:03.092+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:03.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:03.092+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:03.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:03.093+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:03.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:02.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:03.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:02.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:03.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:02.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:03.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:02.852+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:03.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:03.011+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:03.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:03.011+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:03.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:03.011+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:03.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:03.011+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:04.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:03.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:04.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:03.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:04.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:03.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:04.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:03.985+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:04.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:03.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:04.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:03.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:04.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:03.859+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:04.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:03.860+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:04.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:04.097+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:04.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:04.097+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:04.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:04.097+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:04.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:04.097+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:05.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:04.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:05.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:04.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:05.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:04.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:05.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:04.893+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:05.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:05.051+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:05.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:05.051+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:05.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:05.051+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:05.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:05.052+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:05.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:05.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:05.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:05.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:05.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:05.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:05.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:05.028+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:06.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:06.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:06.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:06.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:06.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:06.095+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:06.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:06.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:06.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:06.068+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:06.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:05.935+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:06.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:05.935+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:06.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:05.935+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:06.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:05.935+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:07.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:06.890+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:07.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:06.890+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:07.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:06.890+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:07.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:06.891+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:07.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:07.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:07.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:07.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:07.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:07.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:07.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:07.058+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:07.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:07.053+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:07.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:07.053+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:07.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:07.053+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:07.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:07.054+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:07.701 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:07.263+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20069 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:08.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:08.013+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:08.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:08.013+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:08.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:08.013+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:08.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:08.014+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:08.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:08.056+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:08.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:08.056+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:08.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:08.056+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:08.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:08.057+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:08.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:07.904+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:08.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:07.904+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:08.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:07.904+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:08.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:07.904+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:09.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:09.060+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:09.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:09.060+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:09.060+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:09.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:09.061+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:08.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:08.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:08.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:09.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:08.917+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:09.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:09.048+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:09.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:09.048+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:09.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:09.048+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:09.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:09.049+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:10.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:10.101+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:10.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:10.101+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:10.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:10.101+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:10.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:10.102+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:10.177 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:10.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:10.178 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:10.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:10.178 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:10.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:10.178 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:10.046+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:10.178 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:09.914+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:10.178 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:09.914+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:10.178 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:09.914+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:10.178 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:09.915+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:10.911+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:10.911+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:11.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:10.911+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:11.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:10.911+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:11.051+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:11.051+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:11.051+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:11.052+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:11.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:11.059+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:11.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:11.059+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:11.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:11.059+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:11.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:11.060+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:12.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:11.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:12.020 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:11.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:12.021 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:11.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:12.021 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:11.917+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:12.279 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:12.265+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20158 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:12.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:12.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:12.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:12.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:12.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:12.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:12.280 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:12.019+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:12.280 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:12.025+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:12.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:12.025+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:12.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:12.025+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:12.281 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:12.026+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:12.956+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:12.956+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:12.956+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:13.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:12.957+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:13.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:12.983+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:13.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:12.983+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:13.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:12.983+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:13.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:12.983+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:13.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:13.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:13.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:13.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:13.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:13.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:13.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:13.031+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:14.076+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:14.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:14.076+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:14.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:14.076+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:14.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:14.077+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:14.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:13.975+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:14.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:13.975+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:14.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:13.975+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:14.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:13.975+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:14.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:13.983+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:14.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:13.983+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:14.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:13.983+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:14.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:13.985+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:15.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:14.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:15.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:14.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:15.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:14.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:15.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:14.950+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:15.451 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:14.993+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:15.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:14.993+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:15.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:14.993+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:15.452 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:14.994+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:15.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:15.064+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:15.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:15.064+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:15.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:15.064+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:15.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:15.064+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:15.932+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:15.932+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:16.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:15.932+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:16.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:15.933+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:16.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:16.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:16.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:16.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:16.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:16.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:16.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:16.088+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:16.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:15.959+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:16.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:15.959+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:16.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:15.959+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:16.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:15.959+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:17.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:16.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:17.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:16.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:17.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:16.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:17.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:16.941+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:17.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:17.110+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:17.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:17.110+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:17.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:17.110+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:17.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:17.111+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:17.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:16.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:17.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:16.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:17.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:16.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:17.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:16.917+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:17.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:17.267+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20250 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:18.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:18.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:18.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:18.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:18.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:18.136+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:18.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:18.137+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:18.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:17.920+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:18.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:17.920+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:18.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:17.920+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:18.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:17.921+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:18.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:17.891+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:18.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:17.891+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:18.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:17.891+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:18.203 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:17.892+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:19.145 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:18.905+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:19.145 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:18.905+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:19.145 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:18.905+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:19.145 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:18.906+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:19.145 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:18.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:19.146 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:18.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:19.146 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:18.880+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:19.146 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:18.881+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:19.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:19.142+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:19.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:19.142+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:19.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:19.142+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:19.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:19.143+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:20.150 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:19.898+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:20.150 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:19.898+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:20.150 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:19.898+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:20.151 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:19.899+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:20.151 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:19.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:20.151 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:19.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:20.151 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:19.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:20.151 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:19.896+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:20.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:20.147+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:20.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:20.147+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:20.147+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:20.149+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:20.916+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:20.916+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:21.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:20.916+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:21.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:20.917+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:21.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:21.139+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:21.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:21.139+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:21.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:21.139+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:21.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:21.141+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:21.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:20.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:21.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:20.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:21.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:20.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:21.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:20.902+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:21.871+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:21.871+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:21.871+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:21.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:21.872+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:21.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:21.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:21.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:21.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:21.908+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:22.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:22.269+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20340 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:22.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:22.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:22.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:22.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:22.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:22.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:22.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:22.154+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:23.122 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:22.906+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:23.122 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:22.906+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:23.122 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:22.906+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:23.123 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:22.907+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:23.123 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:22.840+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:23.123 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:22.840+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:23.123 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:22.840+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:23.123 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:22.841+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:23.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:23.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:23.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:23.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:23.119+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:23.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:23.121+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:23.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:23.818+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:23.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:23.818+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:23.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:23.818+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:23.869 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:23.819+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:24.127 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:23.866+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:24.127 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:23.866+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:24.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:23.866+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:24.128 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:23.867+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:24.125+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:24.125+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:24.125+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:24.126+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:25.082 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:24.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:25.082 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:24.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:25.083 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:24.830+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:25.083 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:24.831+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:25.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:24.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:25.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:24.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:25.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:24.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:25.083 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:24.862+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:25.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:25.080+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:25.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:25.080+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:25.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:25.080+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:25.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:25.080+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:26.122 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:25.841+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:26.123 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:25.841+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:26.123 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:25.841+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:26.123 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:25.843+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:26.123 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:25.865+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:26.123 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:25.865+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:26.123 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:25.865+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:26.124 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:25.866+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:26.120+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:26.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:26.120+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:26.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:26.120+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:26.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:26.121+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:27.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:26.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:27.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:26.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:27.080 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:26.831+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:27.081 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:26.833+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:27.081 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:26.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:27.081 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:26.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:27.081 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:26.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:27.081 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:26.824+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:27.399 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:27.272+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20431 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:27.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:27.077+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:27.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:27.077+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:27.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:27.077+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:27.399 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:27.079+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:28.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:27.795+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:28.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:27.795+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:28.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:27.795+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:28.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:27.796+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:28.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:27.776+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:28.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:27.776+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:28.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:27.776+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:28.073 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:27.777+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:28.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:28.069+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:28.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:28.069+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:28.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:28.069+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:28.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:28.070+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:29.094 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:28.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:29.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:28.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:29.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:28.778+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:29.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:28.779+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:29.095 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:28.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:29.095 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:28.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:29.095 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:28.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:29.096 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:28.777+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:29.092+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:29.092+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:29.092+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:29.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:29.093+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:30.077 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:29.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:30.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:29.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:30.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:29.814+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:30.078 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:29.815+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:30.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:29.767+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:30.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:29.767+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:30.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:29.767+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:30.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:29.768+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:30.074+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:30.074+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:30.074+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:30.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:30.075+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:31.045 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:30.833+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:31.045 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:30.833+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:31.045 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:30.833+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:31.045 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:30.835+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:31.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:30.759+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:31.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:30.759+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:31.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:30.759+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:31.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:30.760+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:31.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:31.042+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:31.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:31.042+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:31.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:31.042+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:31.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:31.043+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:32.095 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:31.807+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:32.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:31.807+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:32.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:31.807+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:32.096 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:31.808+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:32.096 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:31.756+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:32.096 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:31.756+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:32.096 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:31.756+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:32.097 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:31.757+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:32.402 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:32.274+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20521 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:32.402 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:32.093+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:32.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:32.093+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:32.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:32.093+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:32.403 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:32.094+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:33.139 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:32.839+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:33.140 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:32.839+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:33.140 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:32.839+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:33.140 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:32.840+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:33.140 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:32.762+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:33.140 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:32.762+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:33.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:32.762+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:33.141 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:32.762+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:33.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:33.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:33.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:33.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:33.138+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:33.885 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:33.731+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:33.886 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:33.731+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:33.886 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:33.731+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:33.886 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:33.732+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:33.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:33.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:33.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:34.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:33.884+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:34.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:34.123+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:34.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:34.123+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:34.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:34.123+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:34.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:34.124+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:34.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:34.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:34.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:34.883+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:34.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:34.884+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:34.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:34.693+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:34.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:34.693+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:34.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:34.693+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:34.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:34.693+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:35.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:35.092+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:35.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:35.092+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:35.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:35.092+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:35.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:35.093+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:35.921+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:35.921+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:35.921+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:35.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:35.922+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:35.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:35.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:35.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:35.679+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:36.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:36.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:36.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:36.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:36.058+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:36.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:36.650+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:36.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:36.650+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:36.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:36.650+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:36.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:36.651+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:36.944+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:36.944+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:36.944+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:37.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:36.944+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:37.026+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:37.026+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:37.026+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:37.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:37.027+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:37.663 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:37.276+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20613 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:37.920 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:37.661+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:37.920 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:37.661+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:37.920 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:37.661+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:37.920 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:37.662+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:37.997+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:37.997+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:37.997+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:37.998+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:37.917+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:37.917+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:37.917+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:37.918+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:38.924 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:38.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:38.924 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:38.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:38.924 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:38.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:38.924 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:38.639+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:38.921+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:38.921+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:39.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:38.921+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:39.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:38.922+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:39.038+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:39.038+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:39.038+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:39.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:39.039+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:39.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:39.884+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:39.885+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:39.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:39.885+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:39.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:39.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:39.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:39.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:39.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:39.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:39.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:39.686+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:40.018+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:40.018+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:40.018+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:40.020+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:40.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:40.900+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:40.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:40.900+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:40.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:40.900+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:40.995 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:40.902+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:40.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:40.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:40.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:40.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:40.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:40.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:40.996 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:40.700+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:40.992+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:40.992+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:40.992+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:40.993+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:41.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:41.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:41.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:41.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:41.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:41.917+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:41.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:41.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:41.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:41.698+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:42.282 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:42.018+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:42.282 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:42.018+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:42.283 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:42.018+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:42.283 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:42.019+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:42.677 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:42.278+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20703 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:42.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:42.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:42.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:42.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:42.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:42.675+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:42.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:42.675+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:42.928 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:42.675+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:42.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:42.676+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:43.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:42.925+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:43.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:42.926+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:43.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:43.052+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:43.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:43.052+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:43.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:43.052+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:43.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:43.052+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:43.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:43.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:43.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:43.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:43.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:43.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:43.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:43.635+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:43.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:43.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:43.897+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:44.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:43.898+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:44.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:44.004+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:44.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:44.004+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:44.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:44.004+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:44.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:44.006+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:44.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:44.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:44.882+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:44.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:44.882+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:44.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:44.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:44.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:44.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:44.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:44.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:44.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:44.656+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:45.046+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:45.046+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:45.046+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:45.047+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:45.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:45.647+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:45.904 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:45.647+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:45.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:45.647+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:45.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:45.647+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:46.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:45.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:45.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:45.902+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:46.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:45.903+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:46.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:46.081+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:46.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:46.081+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:46.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:46.081+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:46.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:46.081+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:46.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:46.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:46.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:46.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:46.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:46.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:46.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:46.697+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:47.283 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:46.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:47.284 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:46.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:47.284 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:46.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:47.284 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:46.953+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:47.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:47.078+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:47.284 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:47.078+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:47.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:47.078+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:47.285 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:47.079+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:47.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:47.281+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20794 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:48.074 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:47.724+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:48.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:47.724+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:48.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:47.724+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:48.075 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:47.724+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:48.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:47.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:48.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:47.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:48.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:47.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:48.076 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:47.956+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:48.071+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:48.072+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:48.072+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:48.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:48.073+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:49.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:48.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:49.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:48.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:49.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:48.962+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:49.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:48.962+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:49.109 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:48.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:49.110 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:48.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:49.110 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:48.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:49.110 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:48.728+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:49.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:49.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:49.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:49.107+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:50.143 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:49.960+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:50.143 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:49.960+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:50.143 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:49.960+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:50.143 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:49.962+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:50.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:49.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:50.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:49.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:50.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:49.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:50.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:49.740+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:50.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:50.140+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:50.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:50.140+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:50.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:50.140+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:50.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:50.141+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:51.151 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:50.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:51.151 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:50.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:51.152 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:50.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:51.152 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:50.936+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:51.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:50.755+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:51.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:50.755+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:51.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:50.755+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:51.152 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:50.755+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:51.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:51.148+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:51.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:51.148+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:51.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:51.148+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:51.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:51.149+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:51.959 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:51.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:51.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:51.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:51.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:51.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:51.960 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:51.737+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:52.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:52.113+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:52.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:52.113+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:52.286 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:52.113+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:52.287 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:52.113+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:52.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:51.957+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:52.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:51.957+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:52.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:51.957+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:52.287 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:51.957+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:52.701 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:52.283+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20884 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:53.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:52.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:53.071 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:52.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:53.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:52.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:53.072 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:52.956+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:53.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:52.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:53.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:52.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:53.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:52.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:53.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:52.742+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:53.069+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:53.069+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:53.069+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:53.070+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:53.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:53.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:53.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:53.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:53.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:53.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:53.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:53.706+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:54.103+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:54.103+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:54.103+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:54.104+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:53.913+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:53.913+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:53.913+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:54.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:53.914+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:55.130 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:54.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:55.130 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:54.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:55.130 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:54.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:55.130 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:54.705+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:55.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:54.934+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:55.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:54.934+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:55.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:54.934+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:55.131 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:54.935+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:55.127+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:55.127+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:55.127+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:55.128+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:56.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:55.891+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:56.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:55.891+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:56.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:55.891+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:56.170 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:55.892+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:56.171 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:55.754+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:56.171 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:55.754+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:56.171 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:55.754+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:56.171 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:55.754+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:56.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:56.167+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:56.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:56.167+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:56.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:56.167+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:56.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:56.169+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:57.126 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:56.751+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:57.127 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:56.751+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:57.127 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:56.751+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:57.127 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:56.751+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:57.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:56.892+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:57.127 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:56.892+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:57.128 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:56.892+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:57.128 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:56.893+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:57.412 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:10:57.285+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 20973 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:10:57.412 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:57.124+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:57.413 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:57.124+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:57.413 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:57.124+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:57.413 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:57.125+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:58.133 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:57.933+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:58.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:57.933+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:58.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:57.933+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:58.134 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:57.933+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:58.135 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:57.765+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:58.135 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:57.765+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:58.135 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:57.765+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:58.136 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:57.766+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:58.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:58.130+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:58.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:58.130+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:58.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:58.130+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:58.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:58.131+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:10:59.110 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:58.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:59.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:58.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:59.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:58.742+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:10:59.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:58.742+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:59.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:58.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:59.111 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:58.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:59.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:58.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:10:59.112 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:58.908+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:10:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:59.108+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:59.108+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:59.108+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:10:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:10:59.110+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:00.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:59.946+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:00.109 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:59.946+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:00.110 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:59.946+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:00.110 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:10:59.946+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:00.110 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:59.731+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:00.110 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:59.731+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:00.110 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:59.731+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:00.110 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:10:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:10:59.732+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:00.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:00.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:00.106+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:00.108+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:00.687+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:00.687+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:00.687+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:00.688+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:01.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:01.107+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:01.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:01.107+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:01.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:01.107+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:01.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:01.108+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:01.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:00.989+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:01.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:00.989+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:01.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:00.989+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:01.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:00.990+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:01.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:01.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:01.729+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:01.730+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:02.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:02.070+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:02.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:02.070+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:02.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:02.070+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:02.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:02.070+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:02.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:01.970+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:02.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:01.970+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:02.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:01.970+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:02.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:01.970+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:02.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:02.287+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21060 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:03.000 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:02.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:03.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:02.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:03.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:02.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:03.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:02.714+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:02.998+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:02.998+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:02.998+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:03.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:02.999+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:03.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:03.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:03.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:03.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:03.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:03.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:03.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:03.028+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:03.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:03.693+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:03.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:03.693+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:03.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:03.693+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:03.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:03.693+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:04.073+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:04.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:04.073+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:04.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:04.073+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:04.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:04.074+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:03.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:03.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:03.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:04.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:03.969+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:04.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:04.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:04.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:04.683+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:05.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:05.059+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:05.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:05.059+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:05.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:05.059+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:05.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:05.060+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:05.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:04.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:05.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:04.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:05.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:04.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:05.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:04.952+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:06.028 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:05.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:06.028 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:05.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:06.028 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:05.940+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:06.028 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:05.941+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:06.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:05.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:06.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:05.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:06.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:05.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:06.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:05.708+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:06.025+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:06.025+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:06.025+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:06.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:06.026+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:06.968 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:06.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:06.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:06.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:06.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:06.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:06.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:06.706+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:07.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:07.040+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:07.292 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:07.040+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:07.293 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:07.040+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:07.293 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:07.041+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:07.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:06.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:07.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:06.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:07.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:06.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:07.293 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:06.967+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:07.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:07.290+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21148 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:07.966 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:07.698+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:07.966 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:07.698+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:07.966 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:07.698+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:07.966 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:07.699+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:08.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:08.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:08.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:08.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:08.008+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:08.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:07.963+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:08.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:07.963+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:08.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:07.963+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:08.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:07.964+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:09.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:08.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:09.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:08.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:09.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:08.915+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:09.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:08.917+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:09.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:08.745+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:09.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:08.745+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:09.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:08.745+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:09.034 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:08.746+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:09.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:09.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:09.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:09.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:09.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:09.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:09.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:09.031+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:10.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:09.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:10.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:09.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:10.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:09.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:10.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:09.958+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:10.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:09.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:10.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:09.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:10.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:09.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:10.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:09.985+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:10.181 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:09.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:10.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:09.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:10.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:09.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:10.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:09.741+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:10.971+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:10.971+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:10.971+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:11.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:10.971+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:11.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:10.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:11.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:10.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:11.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:10.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:11.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:10.968+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:11.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:10.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:11.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:10.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:11.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:10.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:11.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:10.757+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:12.017 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:11.922+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:12.017 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:11.922+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:12.017 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:11.922+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:12.017 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:11.923+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:12.017 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:11.766+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:12.018 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:11.766+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:12.018 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:11.766+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:12.018 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:11.767+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:12.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:12.014+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:12.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:12.014+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:12.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:12.014+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:12.295 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:12.015+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:12.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:12.292+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21229 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:12.971+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:12.971+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:13.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:12.971+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:13.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:12.972+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:13.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:12.905+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:13.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:12.905+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:13.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:12.905+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:13.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:12.907+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:13.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:12.755+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:13.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:12.755+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:13.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:12.755+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:13.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:12.756+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:13.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:13.778+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:13.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:13.778+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:13.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:13.778+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:13.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:13.779+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:14.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:13.887+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:14.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:13.887+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:14.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:13.887+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:14.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:13.889+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:13.948+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:13.948+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:13.948+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:14.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:13.948+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:14.800+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:15.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:14.800+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:15.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:14.800+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:15.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:14.801+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:14.985+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:14.985+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:14.985+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:15.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:14.986+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:15.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:14.877+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:15.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:14.877+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:15.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:14.877+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:15.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:14.878+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:16.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:16.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:16.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:16.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:16.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:16.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:16.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:16.036+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:16.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:15.793+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:16.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:15.793+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:16.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:15.793+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:16.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:15.793+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:16.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:15.842+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:16.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:15.842+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:16.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:15.842+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:16.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:15.843+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:17.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:17.001+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:17.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:17.001+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:17.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:17.001+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:17.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:17.002+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:17.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:16.758+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:17.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:16.758+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:17.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:16.758+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:17.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:16.759+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:17.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:16.817+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:17.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:16.817+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:17.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:16.817+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:17.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:16.818+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:17.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:17.294+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21312 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:18.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:18.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:18.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:18.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:18.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:18.018+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:18.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:17.804+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:18.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:17.804+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:18.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:17.804+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:18.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:17.805+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:18.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:17.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:18.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:17.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:18.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:17.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:18.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:17.785+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:19.037+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:19.037+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:19.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:19.037+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:19.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:19.038+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:19.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:18.843+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:19.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:18.843+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:19.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:18.843+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:19.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:18.843+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:19.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:18.772+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:19.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:18.772+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:19.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:18.772+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:19.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:18.773+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:20.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:20.003+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:20.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:20.003+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:20.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:20.003+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:20.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:20.004+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:20.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:19.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:20.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:19.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:20.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:19.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:20.183 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:19.865+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:20.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:19.756+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:20.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:19.756+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:20.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:19.756+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:20.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:19.756+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:21.000 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:20.721+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:21.000 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:20.721+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:21.000 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:20.721+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:21.000 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:20.722+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:21.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:20.835+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:21.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:20.835+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:21.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:20.835+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:21.001 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:20.836+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:20.997+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:21.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:20.997+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:21.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:20.997+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:21.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:20.998+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:21.979 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:21.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:21.979 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:21.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:21.979 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:21.708+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:21.979 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:21.709+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:21.979 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:21.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:21.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:21.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:21.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:21.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:21.980 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:21.831+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:22.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:21.976+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:22.299 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:21.976+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:22.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:21.976+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:22.300 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:21.977+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:22.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:22.297+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21396 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:23.011 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:22.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:23.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:22.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:23.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:22.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:23.012 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:22.741+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:23.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:22.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:23.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:22.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:23.012 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:22.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:23.013 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:22.789+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:23.009+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:23.009+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:23.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:23.009+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:23.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:23.010+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:23.766+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:23.766+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:23.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:23.766+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:23.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:23.767+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:23.811+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:23.811+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:23.811+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:23.812+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:23.996+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:23.996+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:23.996+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:24.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:23.997+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:24.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:24.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:25.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:24.956+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:25.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:24.957+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:24.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:24.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:24.746+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:25.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:24.748+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:25.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:24.775+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:25.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:24.775+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:25.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:24.775+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:25.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:24.776+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:26.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:25.777+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:26.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:25.777+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:26.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:25.777+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:26.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:25.779+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:26.191 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:25.819+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:26.192 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:25.819+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:26.192 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:25.819+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:26.192 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:25.820+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:26.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:25.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:26.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:25.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:26.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:25.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:26.192 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:25.936+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:27.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:26.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:27.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:26.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:27.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:26.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:27.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:26.783+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:27.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:26.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:27.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:26.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:27.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:26.734+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:27.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:26.735+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:27.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:26.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:27.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:26.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:27.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:26.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:27.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:26.982+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:27.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:27.299+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21481 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:27.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:27.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:27.738+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:27.739+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:28.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:27.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:28.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:27.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:28.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:27.937+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:28.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:27.938+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:28.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:27.778+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:28.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:27.778+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:28.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:27.778+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:28.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:27.779+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:29.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:28.781+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:29.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:28.781+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:29.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:28.781+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:29.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:28.781+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:29.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:28.926+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:29.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:28.926+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:29.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:28.926+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:29.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:28.927+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:29.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:28.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:29.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:28.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:29.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:28.785+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:29.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:28.785+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:30.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:29.941+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:30.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:29.941+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:30.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:29.941+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:30.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:29.942+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:30.182 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:29.818+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:30.182 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:29.818+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:30.182 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:29.818+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:30.182 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:29.819+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:30.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:29.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:30.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:29.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:30.182 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:29.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:30.183 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:29.749+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:30.794+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:30.794+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:30.794+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:30.794+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:31.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:30.787+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:31.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:30.787+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:31.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:30.787+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:31.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:30.788+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:31.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:30.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:31.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:30.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:31.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:30.907+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:31.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:30.908+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:32.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:31.788+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:32.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:31.788+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:32.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:31.788+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:32.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:31.789+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:32.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:31.904+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:32.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:31.904+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:32.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:31.904+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:32.202 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:31.905+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:32.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:31.779+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:32.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:31.779+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:32.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:31.779+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:32.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:31.780+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:32.701 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:32.302+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21571 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:32.926+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:32.926+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:32.926+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:33.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:32.927+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:33.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:32.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:33.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:32.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:33.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:32.770+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:33.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:32.771+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:33.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:32.773+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:33.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:32.773+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:33.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:32.773+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:33.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:32.774+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:33.771+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:33.771+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:33.771+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:33.772+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:33.808+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:33.808+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:33.808+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:33.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:33.809+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:33.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:33.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:33.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:34.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:33.959+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:34.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:35.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:34.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:34.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:35.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:34.929+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:35.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:34.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:35.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:34.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:35.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:34.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:35.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:34.778+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:35.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:34.732+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:35.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:34.732+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:35.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:34.732+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:35.202 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:34.733+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:35.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:35.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:35.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:35.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:35.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:35.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:35.971 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:35.709+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:35.971 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:35.767+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:35.767+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:35.767+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:35.972 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:35.768+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:35.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:35.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:35.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:35.969+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:36.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:36.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:36.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:36.662+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:36.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:36.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:36.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:36.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:36.757+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:37.306 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:36.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:37.307 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:36.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:37.307 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:36.958+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:37.307 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:36.959+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:37.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:37.304+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21663 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:37.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:37.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:38.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:37.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:38.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:37.929+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:37.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:37.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:37.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:38.201 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:37.704+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:38.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:37.791+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:38.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:37.791+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:38.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:37.791+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:38.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:37.792+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:38.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:38.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:38.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:38.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:38.924+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:38.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:38.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:38.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:38.679+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:38.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:38.836+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:38.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:38.836+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:38.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:38.836+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:38.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:38.837+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:39.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:39.805+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:39.930 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:39.805+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:39.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:39.805+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:39.931 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:39.806+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:39.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:39.633+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:39.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:39.633+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:39.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:39.633+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:39.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:39.633+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:40.181 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:39.927+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:40.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:39.927+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:40.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:39.927+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:40.182 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:39.928+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:40.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:40.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:40.933 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:40.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:40.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:40.768+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:40.934 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:40.769+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:40.934 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:40.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:40.934 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:40.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:40.934 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:40.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:40.934 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:40.659+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:40.931+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:40.931+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:41.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:40.931+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:41.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:40.932+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:41.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:41.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:41.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:41.775+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:41.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:41.776+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:41.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:41.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:41.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:41.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:41.679+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:42.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:41.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:42.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:41.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:42.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:41.966+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:42.310 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:41.967+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:42.684 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:42.306+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21752 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:42.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:42.809+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:42.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:42.809+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:42.809+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:42.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:42.810+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:42.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:42.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:42.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:42.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:42.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:42.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:42.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:42.682+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:43.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:42.987+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:43.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:42.987+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:43.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:42.987+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:43.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:42.988+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:43.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:43.839+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:43.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:43.839+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:43.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:43.839+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:43.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:43.840+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:43.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:43.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:43.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:43.665+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:44.022+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:44.022+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:44.022+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:44.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:44.023+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:44.876 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:44.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:44.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:44.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:44.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:44.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:44.877 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:44.679+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:44.874+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:44.874+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:45.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:44.874+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:45.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:44.875+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:45.036+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:45.036+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:45.036+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:45.201 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:45.037+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:45.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:45.915+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:45.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:45.915+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:45.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:45.915+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:45.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:45.915+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:45.695+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:45.695+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:45.695+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:45.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:45.695+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:46.037+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:46.037+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:46.037+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:46.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:46.038+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:47.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:46.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:47.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:46.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:47.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:46.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:47.029 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:46.719+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:47.029 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:46.919+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:47.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:46.919+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:47.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:46.919+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:47.033 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:46.920+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:47.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:47.026+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:47.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:47.026+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:47.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:47.026+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:47.311 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:47.027+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:47.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:47.309+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21843 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:47.983 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:47.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:47.983 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:47.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:47.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:47.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:47.984 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:47.957+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:47.984 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:47.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:47.984 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:47.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:47.984 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:47.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:47.984 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:47.711+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:47.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:47.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:47.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:47.981+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:49.010 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:48.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:49.010 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:48.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:49.010 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:48.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:49.010 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:48.743+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:49.010 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:48.974+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:49.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:48.974+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:49.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:48.974+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:49.011 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:48.975+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:49.007+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:49.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:49.007+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:49.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:49.007+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:49.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:49.008+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:50.021 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:49.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:50.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:49.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:50.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:49.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:50.022 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:49.720+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:50.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:50.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:50.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:50.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:50.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:50.018+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:50.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:50.019+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:50.035+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:50.035+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:50.035+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:50.037+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:51.071 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:50.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:51.071 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:50.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:51.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:50.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:51.072 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:50.728+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:51.077+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:51.077+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:51.077+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:51.077+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:51.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:51.069+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:51.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:51.069+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:51.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:51.069+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:51.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:51.069+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:52.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:51.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:52.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:51.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:52.051 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:51.777+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:52.052 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:51.777+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:52.314 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:52.083+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:52.314 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:52.083+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:52.315 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:52.083+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:52.315 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:52.084+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:52.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:52.048+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:52.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:52.048+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:52.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:52.048+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:52.315 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:52.049+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:52.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:52.311+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 21932 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:53.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:52.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:53.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:52.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:53.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:52.748+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:53.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:52.748+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:53.043+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:53.043+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:53.043+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:53.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:53.044+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:53.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:53.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:53.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:53.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:53.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:53.080+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:53.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:53.081+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:53.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:53.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:53.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:53.720+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:54.070+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:54.070+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:54.070+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:54.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:54.071+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:54.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:54.062+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:54.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:54.062+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:54.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:54.062+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:54.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:54.063+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:54.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:54.695+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:54.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:54.695+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:54.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:54.695+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:54.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:54.695+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:55.028+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:55.028+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:55.028+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:55.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:55.028+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:55.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:55.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:55.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:55.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:55.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:55.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:55.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:55.044+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:56.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:55.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:56.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:55.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:56.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:55.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:56.080 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:55.727+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:56.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:56.081+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:56.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:56.081+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:56.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:56.081+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:56.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:56.083+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:56.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:56.076+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:56.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:56.076+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:56.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:56.076+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:56.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:56.078+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:57.054 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:56.769+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:57.054 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:56.769+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:57.054 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:56.769+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:57.054 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:56.769+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:57.317 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:57.051+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:57.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:57.051+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:57.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:57.051+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:57.318 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:57.052+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:57.318 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:57.095+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:57.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:57.095+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:57.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:57.095+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:57.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:57.096+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:57.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:11:57.314+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22024 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:11:58.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:57.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:58.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:57.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:58.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:57.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:58.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:57.743+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:58.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:58.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:58.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:58.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:58.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:58.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:58.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:58.083+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:58.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:58.075+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:58.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:58.075+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:58.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:58.075+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:58.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:58.077+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:59.049 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:58.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:59.049 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:58.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:59.049 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:58.783+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:11:59.049 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:58.783+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:11:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:59.120+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:59.120+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:59.120+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:11:59.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:11:59.120+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:11:59.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:59.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:59.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:59.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:59.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:59.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:11:59.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:11:59.048+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:00.040 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:59.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:00.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:59.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:00.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:59.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:00.041 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:11:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:11:59.739+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:00.151+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:00.151+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:00.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:00.151+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:00.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:00.152+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:00.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:00.038+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:00.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:00.038+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:00.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:00.038+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:00.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:00.039+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:01.027 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:01.024+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:01.027 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:01.024+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:01.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:00.774+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:01.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:00.774+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:01.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:00.774+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:01.028 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:00.774+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:01.024+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:01.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:01.025+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:01.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:01.199+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:01.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:01.199+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:01.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:01.199+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:01.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:01.200+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:02.013 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:01.747+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:02.014 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:01.747+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:02.014 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:01.747+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:02.014 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:01.747+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:02.319 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:02.214+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:02.319 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:02.214+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:02.319 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:02.214+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:02.319 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:02.215+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:02.319 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:02.011+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:02.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:02.011+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:02.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:02.011+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:02.320 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:02.012+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:02.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:02.316+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22113 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:03.047 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:02.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:03.047 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:02.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:03.047 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:02.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:03.047 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:02.704+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:03.238+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:03.238+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:03.238+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:03.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:03.239+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:03.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:03.044+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:03.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:03.044+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:03.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:03.044+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:03.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:03.045+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:03.731+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:03.731+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:03.731+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:03.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:03.731+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:04.235+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:04.235+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:04.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:04.235+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:04.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:04.236+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:04.036+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:04.036+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:04.036+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:04.037+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:05.005 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:04.717+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:05.006 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:04.717+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:05.006 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:04.717+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:05.006 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:04.718+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:05.003+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:05.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:05.003+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:05.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:05.003+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:05.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:05.004+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:05.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:05.194+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:05.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:05.194+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:05.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:05.194+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:05.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:05.194+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:05.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:05.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:05.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:05.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:05.698+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:06.008+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:06.008+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:06.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:06.008+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:06.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:06.008+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:06.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:06.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:06.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:06.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:06.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:06.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:06.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:06.166+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:07.024 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:06.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:07.024 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:06.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:07.024 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:06.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:07.024 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:06.718+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:07.321 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:07.195+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:07.321 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:07.195+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:07.321 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:07.195+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:07.322 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:07.196+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:07.322 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:07.021+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:07.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:07.021+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:07.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:07.021+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:07.323 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:07.022+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:07.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:07.318+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22204 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:08.042 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:07.740+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:08.042 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:07.740+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:08.043 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:07.740+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:08.043 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:07.740+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:08.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:08.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:08.040+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:08.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:08.040+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:08.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:08.041+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:08.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:08.203+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:08.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:08.203+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:08.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:08.203+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:08.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:08.204+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:09.032 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:08.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:09.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:08.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:09.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:08.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:09.033 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:08.700+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:09.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:09.208+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:09.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:09.208+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:09.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:09.208+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:09.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:09.209+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:09.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:09.029+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:09.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:09.029+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:09.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:09.029+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:09.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:09.030+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:09.653+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:09.653+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:09.653+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:09.653+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:09.986+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:09.986+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:09.986+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:10.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:09.987+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:10.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:10.176+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:10.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:10.176+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:10.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:10.176+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:10.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:10.177+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:10.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:10.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:10.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:10.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:10.662+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:11.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:11.223+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:11.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:11.223+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:11.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:11.223+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:11.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:11.224+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:11.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:10.988+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:11.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:10.988+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:11.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:10.988+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:11.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:10.989+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:11.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:11.629+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:11.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:11.629+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:11.629+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:11.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:11.630+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:12.270 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:12.228+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:12.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:12.228+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:12.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:12.228+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:12.271 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:12.229+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:12.271 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:11.999+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:12.272 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:11.999+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:12.272 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:11.999+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:12.272 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:12.000+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:12.587 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:12.321+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22294 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:12.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:12.584+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:12.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:12.584+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:12.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:12.584+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:12.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:12.585+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:12.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:12.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:13.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:12.968+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:13.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:12.969+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:13.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:13.185+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:13.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:13.185+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:13.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:13.185+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:13.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:13.186+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:13.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:13.632+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:13.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:13.632+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:13.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:13.632+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:13.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:13.632+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:14.158+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:14.158+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:14.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:14.158+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:14.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:14.159+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:14.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:13.949+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:14.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:13.949+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:14.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:13.949+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:14.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:13.949+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:14.623+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:14.623+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:14.623+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:14.624+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:15.342 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:15.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:15.343 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:15.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:15.343 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:15.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:15.343 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:15.158+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:15.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:14.984+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:15.343 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:14.984+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:15.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:14.984+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:15.344 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:14.984+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:15.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:15.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:15.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:15.649+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:16.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:16.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:16.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:16.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:16.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:16.202+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:16.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:16.203+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:16.008+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:16.008+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:16.008+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:16.009+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:16.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:16.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:16.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:16.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:16.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:16.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:16.664+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:17.326 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:16.985+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:17.326 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:16.985+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:17.326 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:16.985+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:17.326 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:16.986+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:17.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:17.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:17.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:17.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:17.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:17.165+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:17.327 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:17.166+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:17.694 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:17.323+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22386 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:17.691+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:17.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:17.691+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:17.691+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:17.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:17.692+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:18.015+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:18.015+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:18.015+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:18.016+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:18.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:18.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:18.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:18.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:18.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:18.137+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:18.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:18.138+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:18.691+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:18.691+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:18.691+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:18.692+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:19.023+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:19.023+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:19.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:19.023+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:19.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:19.024+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:19.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:19.139+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:19.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:19.139+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:19.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:19.139+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:19.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:19.140+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:19.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:19.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:19.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:19.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:19.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:19.658+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:20.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:20.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:20.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:20.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:20.153+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:20.154+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:20.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:20.007+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:20.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:20.007+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:20.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:20.007+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:20.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:20.008+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:20.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:20.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:20.694+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:20.695+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:21.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:21.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:21.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:21.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:21.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:21.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:21.016+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:21.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:21.186+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:21.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:21.186+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:21.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:21.187+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:21.454 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:21.188+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:21.687+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:21.687+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:21.687+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:21.688+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:22.328 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:22.167+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:22.329 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:22.167+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:22.329 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:22.167+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:22.329 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:22.168+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:22.329 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:22.063+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:22.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:22.063+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:22.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:22.063+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:22.330 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:22.064+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:22.650 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:22.326+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22476 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:22.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:22.647+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:22.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:22.647+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:22.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:22.647+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:22.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:22.648+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:23.163+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:23.163+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:23.163+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:23.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:23.163+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:23.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:23.068+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:23.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:23.068+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:23.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:23.068+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:23.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:23.070+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:23.604+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:23.604+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:23.604+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:23.605+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:24.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:24.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:24.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:24.157+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:24.158+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:24.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:24.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:24.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:24.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:24.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:24.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:24.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:24.055+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:24.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:24.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:24.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:24.620+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:25.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:25.123+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:25.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:25.123+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:25.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:25.123+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:25.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:25.124+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:25.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:25.041+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:25.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:25.041+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:25.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:25.041+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:25.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:25.041+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:25.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:25.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:25.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:25.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:25.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:25.647+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:26.077+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:26.077+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:26.077+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:26.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:26.077+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:26.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:26.071+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:26.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:26.071+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:26.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:26.071+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:26.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:26.072+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:26.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:26.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:26.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:26.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:26.664+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:27.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:27.047+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:27.331 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:27.047+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:27.332 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:27.047+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:27.332 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:27.048+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:27.332 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:27.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:27.332 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:27.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:27.332 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:27.082+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:27.332 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:27.083+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:27.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:27.328+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22567 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:28.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:27.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:28.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:27.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:28.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:27.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:28.046 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:27.706+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:28.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:28.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:28.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:28.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:28.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:28.043+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:28.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:28.044+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:28.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:28.075+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:28.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:28.075+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:28.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:28.075+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:28.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:28.075+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:28.670+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:28.670+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:28.670+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:28.671+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:29.025+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:29.025+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:29.025+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:29.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:29.026+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:29.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:29.000+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:29.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:29.000+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:29.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:29.000+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:29.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:29.001+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:29.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:29.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:29.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:29.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:29.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:29.682+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:29.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:29.682+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:30.037+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:30.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:30.037+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:30.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:30.037+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:30.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:30.038+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:30.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:30.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:30.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:30.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:30.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:30.035+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:30.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:30.036+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:31.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:30.732+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:31.069 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:30.732+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:31.070 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:30.732+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:31.070 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:30.733+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:31.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:31.081+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:31.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:31.081+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:31.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:31.081+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:31.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:31.081+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:31.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:31.065+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:31.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:31.065+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:31.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:31.065+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:31.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:31.066+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:31.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:31.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:31.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:31.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:31.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:31.689+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:32.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:32.105+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:32.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:32.105+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:32.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:32.105+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:32.334 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:32.105+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:32.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:32.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:32.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:32.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:32.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:32.054+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:32.335 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:32.054+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:32.649 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:32.331+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22657 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:32.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:32.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:32.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:32.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:32.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:32.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:32.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:32.647+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:33.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:33.031+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:33.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:33.031+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:33.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:33.031+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:33.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:33.032+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:33.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:33.063+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:33.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:33.063+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:33.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:33.063+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:33.453 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:33.064+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:33.642+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:33.642+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:33.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:33.642+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:33.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:33.642+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:34.056+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:34.056+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:34.056+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:34.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:34.056+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:34.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:34.095+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:34.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:34.095+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:34.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:34.095+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:34.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:34.096+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:34.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:34.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:34.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:34.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:34.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:34.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:34.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:34.622+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:35.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:35.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:35.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:35.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:35.026+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:35.146+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:35.146+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:35.146+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:35.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:35.146+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:35.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:35.572+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:35.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:35.572+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:35.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:35.572+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:35.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:35.573+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:36.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:36.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:36.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:36.058+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:36.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:36.059+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:36.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:36.174+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:36.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:36.174+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:36.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:36.174+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:36.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:36.175+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:36.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:36.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:36.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:36.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:36.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:36.548+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:37.360 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:37.333+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22749 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:37.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:37.103+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:37.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:37.103+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:37.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:37.103+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:37.361 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:37.103+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:37.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:37.188+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:37.362 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:37.188+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:37.362 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:37.188+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:37.362 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:37.189+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:37.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:37.596+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:37.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:37.596+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:37.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:37.596+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:37.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:37.597+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:38.190+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:38.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:38.190+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:38.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:38.190+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:38.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:38.191+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:38.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:38.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:38.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:38.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:38.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:38.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:38.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:38.116+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:38.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:38.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:38.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:38.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:38.589+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:39.112+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:39.112+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:39.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:39.112+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:39.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:39.113+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:39.212+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:39.213+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:39.213+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:39.213+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:39.605+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:39.605+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:39.605+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:39.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:39.606+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:40.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:40.090+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:40.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:40.090+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:40.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:40.090+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:40.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:40.090+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:40.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:40.256+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:40.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:40.256+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:40.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:40.256+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:40.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:40.257+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:40.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:40.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:40.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:40.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:40.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:40.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:40.615+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:41.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:41.229+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:41.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:41.229+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:41.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:41.229+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:41.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:41.230+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:41.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:41.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:41.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:41.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:41.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:41.045+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:41.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:41.046+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:41.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:41.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:41.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:41.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:41.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:41.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:41.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:41.663+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:42.338 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:42.184+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:42.184+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:42.184+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:42.339 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:42.185+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:42.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:42.019+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:42.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:42.019+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:42.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:42.019+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:42.340 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:42.019+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:42.641 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:42.336+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22839 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:42.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:42.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:42.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:42.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:42.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:42.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:42.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:42.639+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:43.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:43.209+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:43.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:43.209+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:43.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:43.209+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:43.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:43.210+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:43.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:43.014+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:43.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:43.014+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:43.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:43.014+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:43.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:43.015+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:43.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:43.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:43.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:43.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:43.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:43.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:43.954 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:43.668+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:44.248 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:43.992+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:44.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:43.992+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:44.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:43.992+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:44.249 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:43.992+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:44.642 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:44.246+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:44.642 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:44.246+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:44.642 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:44.246+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:44.642 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:44.248+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:44.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:44.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:44.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:44.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:44.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:44.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:44.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:44.640+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:45.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:45.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:45.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:45.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:45.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:45.025+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:45.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:45.025+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:45.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:45.204+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:45.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:45.204+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:45.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:45.204+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:45.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:45.204+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:45.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:45.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:45.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:45.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:45.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:45.681+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:46.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:46.070+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:46.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:46.070+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:46.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:46.070+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:46.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:46.070+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:46.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:46.254+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:46.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:46.254+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:46.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:46.254+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:46.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:46.255+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:47.104 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:46.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:47.105 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:46.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:47.105 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:46.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:47.105 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:46.713+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:47.363 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:47.337+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 22930 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:47.364 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:47.100+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:47.364 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:47.100+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:47.364 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:47.100+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:47.364 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:47.101+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:47.364 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:47.252+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:47.365 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:47.252+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:47.365 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:47.252+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:47.365 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:47.253+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:48.070 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:47.725+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:48.070 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:47.725+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:48.070 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:47.725+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:48.070 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:47.725+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:48.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:48.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:48.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:48.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:48.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:48.068+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:48.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:48.227+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:48.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:48.227+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:48.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:48.227+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:48.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:48.227+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:49.110 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:48.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:49.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:48.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:49.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:48.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:49.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:48.726+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:49.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:49.108+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:49.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:49.108+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:49.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:49.108+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:49.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:49.109+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:49.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:49.244+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:49.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:49.244+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:49.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:49.244+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:49.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:49.245+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:49.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:49.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:49.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:49.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:49.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:49.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:49.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:49.683+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:50.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:50.070+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:50.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:50.070+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:50.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:50.070+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:50.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:50.071+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:50.195+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:50.195+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:50.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:50.195+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:50.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:50.196+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:51.081 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:50.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:51.081 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:50.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:51.082 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:50.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:51.082 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:50.703+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:51.235+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:51.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:51.235+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:51.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:51.235+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:51.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:51.236+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:51.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:51.079+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:51.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:51.079+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:51.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:51.079+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:51.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:51.080+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:51.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:51.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:51.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:51.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:51.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:51.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:51.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:51.659+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:52.346 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:52.086+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:52.346 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:52.086+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:52.346 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:52.086+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:52.346 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:52.087+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:52.347 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:52.282+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:52.347 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:52.282+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:52.347 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:52.282+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:52.347 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:52.284+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:52.677 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:52.341+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23020 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:52.675+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:52.675+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:52.675+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:52.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:52.675+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:53.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:53.243+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:53.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:53.243+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:53.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:53.243+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:53.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:53.244+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:53.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:53.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:53.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:53.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:53.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:53.046+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:53.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:53.046+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:53.674+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:53.674+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:53.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:53.674+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:53.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:53.674+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:54.255+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:54.255+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:54.255+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:54.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:54.255+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:54.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:54.059+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:54.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:54.059+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:54.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:54.059+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:54.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:54.060+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:54.714 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:54.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:54.714 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:54.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:54.714 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:54.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:54.714 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:54.680+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:55.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:55.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:55.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:55.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:55.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:55.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:55.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:55.053+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:55.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:55.274+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:55.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:55.274+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:55.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:55.274+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:55.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:55.275+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:55.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:55.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:55.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:55.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:55.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:55.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:55.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:55.635+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:56.453 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:56.087+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:56.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:56.087+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:56.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:56.087+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:56.454 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:56.087+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:56.455 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:56.297+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:56.455 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:56.297+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:56.455 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:56.297+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:56.455 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:56.298+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:56.656+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:56.656+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:56.656+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:56.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:56.657+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:57.296 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:57.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:57.296 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:57.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:57.297 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:57.039+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:57.297 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:57.040+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:57.662 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:12:57.343+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23112 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:12:57.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:57.294+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:57.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:57.294+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:57.663 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:57.294+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:57.664 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:57.295+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:57.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:57.659+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:57.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:57.659+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:57.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:57.659+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:57.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:57.660+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:58.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:58.305+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:58.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:58.305+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:58.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:58.305+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:58.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:58.306+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:58.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:58.068+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:58.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:58.068+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:58.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:58.068+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:58.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:58.068+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:58.619+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:58.619+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:58.619+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:58.620+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:59.099+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:59.099+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:59.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:59.099+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:12:59.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:12:59.100+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:12:59.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:59.292+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:59.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:59.292+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:59.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:59.292+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:12:59.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:12:59.293+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:12:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:59.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:59.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:59.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:59.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:59.663+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:12:59.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:12:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:12:59.664+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:00.147+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:00.147+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:00.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:00.147+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:00.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:00.148+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:00.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:00.256+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:00.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:00.256+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:00.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:00.256+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:00.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:00.257+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:00.617+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:00.617+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:00.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:00.617+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:00.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:00.618+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:01.303 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:01.191+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:01.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:01.191+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:01.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:01.191+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:01.304 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:01.192+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:01.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:01.300+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:01.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:01.300+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:01.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:01.300+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:01.653 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:01.302+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:01.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:01.650+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:01.650+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:01.650+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:01.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:01.651+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:02.476 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:02.346+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23202 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:02.476 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:02.291+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:02.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:02.291+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:02.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:02.291+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:02.477 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:02.293+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:02.477 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:02.219+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:02.477 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:02.219+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:02.477 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:02.219+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:02.477 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:02.220+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:02.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:02.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:02.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:02.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:02.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:02.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:02.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:02.683+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:03.621 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:03.199+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:03.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:03.199+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:03.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:03.199+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:03.622 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:03.201+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:03.623 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:03.278+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:03.623 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:03.278+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:03.623 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:03.278+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:03.623 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:03.278+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:03.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:03.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:03.662+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:03.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:03.663+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:04.177+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:04.177+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:04.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:04.177+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:04.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:04.177+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:04.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:04.269+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:04.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:04.269+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:04.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:04.269+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:04.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:04.270+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:04.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:04.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:04.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:04.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:04.615+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:05.171+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:05.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:05.171+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:05.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:05.171+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:05.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:05.172+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:05.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:05.278+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:05.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:05.278+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:05.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:05.278+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:05.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:05.280+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:05.645+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:05.645+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:05.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:05.645+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:05.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:05.646+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:06.248+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:06.248+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:06.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:06.248+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:06.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:06.249+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:06.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:06.168+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:06.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:06.168+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:06.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:06.168+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:06.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:06.168+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:06.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:06.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:06.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:06.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:06.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:06.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:06.685+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:07.450 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:07.348+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23293 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:07.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:07.239+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:07.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:07.239+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:07.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:07.239+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:07.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:07.241+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:07.126+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:07.126+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:07.126+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:07.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:07.128+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:07.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:07.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:07.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:07.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:07.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:07.646+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:07.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:07.647+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:08.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:08.192+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:08.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:08.192+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:08.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:08.192+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:08.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:08.193+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:08.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:08.167+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:08.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:08.167+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:08.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:08.167+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:08.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:08.168+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:08.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:08.598+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:08.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:08.598+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:08.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:08.598+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:08.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:08.599+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:09.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:09.160+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:09.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:09.160+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:09.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:09.160+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:09.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:09.161+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:09.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:09.203+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:09.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:09.203+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:09.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:09.203+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:09.452 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:09.203+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:09.608+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:09.608+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:09.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:09.608+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:09.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:09.609+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:10.150+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:10.151+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:10.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:10.151+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:10.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:10.152+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:10.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:10.175+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:10.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:10.175+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:10.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:10.175+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:10.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:10.176+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:10.595+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:10.595+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:10.595+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:10.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:10.595+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:11.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:11.185+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:11.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:11.185+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:11.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:11.185+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:11.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:11.186+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:11.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:11.132+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:11.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:11.132+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:11.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:11.132+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:11.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:11.134+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:11.719 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:11.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:11.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:11.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:11.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:11.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:11.720 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:11.586+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:12.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:12.169+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:12.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:12.169+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:12.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:12.169+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:12.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:12.170+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:12.355 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:12.156+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:12.355 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:12.156+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:12.355 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:12.156+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:12.355 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:12.157+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:12.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:12.351+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23383 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:12.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:12.568+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:12.568+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:12.568+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:12.568+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:13.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:13.146+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:13.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:13.146+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:13.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:13.146+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:13.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:13.148+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:13.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:13.169+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:13.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:13.169+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:13.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:13.169+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:13.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:13.170+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:13.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:13.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:13.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:13.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:13.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:13.614+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:13.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:13.614+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:14.154+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:14.154+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:14.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:14.154+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:14.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:14.155+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:14.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:14.127+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:14.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:14.127+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:14.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:14.127+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:14.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:14.128+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:14.565+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:14.565+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:14.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:14.565+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:14.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:14.566+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:15.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:15.127+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:15.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:15.127+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:15.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:15.127+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:15.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:15.127+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:15.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:15.115+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:15.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:15.115+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:15.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:15.115+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:15.452 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:15.116+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:15.561+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:15.561+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:15.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:15.561+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:15.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:15.561+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:16.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:16.083+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:16.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:16.083+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:16.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:16.083+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:16.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:16.083+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:16.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:16.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:16.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:16.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:16.158+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:16.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:16.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:16.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:16.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:16.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:16.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:16.551+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:17.356 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:17.062+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:17.356 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:17.062+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:17.356 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:17.062+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:17.357 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:17.063+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:17.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:17.203+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:17.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:17.203+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:17.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:17.203+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:17.357 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:17.204+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:17.702 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:17.353+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23475 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:17.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:17.576+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:17.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:17.576+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:17.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:17.576+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:17.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:17.576+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:18.248+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:18.248+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:18.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:18.248+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:18.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:18.248+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:18.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:18.024+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:18.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:18.024+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:18.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:18.024+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:18.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:18.026+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:18.549+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:18.549+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:18.549+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:18.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:18.550+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:19.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:19.024+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:19.300 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:19.024+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:19.301 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:19.024+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:19.301 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:19.025+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:19.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:19.298+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:19.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:19.298+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:19.602 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:19.298+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:19.603 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:19.299+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:19.600+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:19.600+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:19.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:19.600+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:19.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:19.600+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:20.273+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:20.273+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:20.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:20.273+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:20.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:20.273+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:20.055+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:20.055+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:20.055+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:20.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:20.056+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:20.563+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:20.563+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:20.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:20.563+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:20.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:20.563+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:21.241+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:21.241+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:21.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:21.241+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:21.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:21.242+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:21.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:21.073+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:21.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:21.073+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:21.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:21.073+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:21.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:21.074+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:21.603+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:21.603+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:21.603+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:21.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:21.603+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:22.359 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:22.037+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:22.359 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:22.037+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:22.359 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:22.037+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:22.359 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:22.038+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:22.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:22.232+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:22.359 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:22.232+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:22.360 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:22.232+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:22.360 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:22.233+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:22.612 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:22.356+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23565 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:22.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:22.610+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:22.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:22.610+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:22.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:22.610+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:22.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:22.610+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:23.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:23.039+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:23.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:23.039+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:23.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:23.039+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:23.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:23.040+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:23.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:23.216+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:23.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:23.216+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:23.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:23.216+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:23.592 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:23.217+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:23.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:23.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:23.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:23.588+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:23.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:23.589+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:24.258+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:24.258+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:24.258+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:24.451 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:24.259+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:24.060+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:24.060+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:24.060+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:24.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:24.061+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:24.559+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:24.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:24.559+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:24.559+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:24.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:24.560+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:25.296 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:25.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:25.296 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:25.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:25.296 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:25.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:25.296 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:25.014+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:25.586 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:25.293+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:25.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:25.293+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:25.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:25.293+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:25.587 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:25.294+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:25.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:25.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:25.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:25.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:25.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:25.584+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:26.263 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:25.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:26.264 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:25.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:26.264 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:25.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:26.264 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:25.968+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:26.605 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:26.261+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:26.606 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:26.261+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:26.606 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:26.261+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:26.606 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:26.262+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:26.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:26.602+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:26.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:26.602+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:26.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:26.603+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:26.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:26.603+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:27.315 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:27.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:27.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:27.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:27.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:27.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:27.316 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:27.005+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:27.581 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:27.358+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23656 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:27.581 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:27.312+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:27.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:27.312+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:27.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:27.312+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:27.582 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:27.313+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:27.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:27.578+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:27.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:27.578+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:27.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:27.578+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:27.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:27.578+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:28.360 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:28.026+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:28.360 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:28.026+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:28.360 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:28.026+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:28.360 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:28.027+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:28.624 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:28.357+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:28.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:28.357+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:28.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:28.357+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:28.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:28.358+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:28.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:28.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:28.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:28.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:28.622+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:29.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:29.014+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:29.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:29.014+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:29.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:29.014+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:29.363 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:29.015+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:29.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:29.360+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:29.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:29.360+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:29.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:29.360+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:29.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:29.361+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:29.593+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:29.593+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:29.593+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:29.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:29.593+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:30.351 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:30.058+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:30.352 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:30.058+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:30.352 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:30.058+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:30.352 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:30.059+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:30.635 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:30.349+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:30.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:30.349+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:30.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:30.349+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:30.636 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:30.350+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:30.630+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:30.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:30.630+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:30.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:30.630+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:30.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:30.631+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:31.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:31.051+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:31.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:31.051+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:31.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:31.051+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:31.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:31.052+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:31.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:31.355+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:31.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:31.355+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:31.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:31.355+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:31.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:31.356+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:31.618+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:31.618+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:31.618+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:31.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:31.618+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:32.364 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:32.021+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:32.364 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:32.021+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:32.364 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:32.021+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:32.364 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:32.022+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:32.629 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:32.361+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23746 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:32.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:32.386+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:32.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:32.386+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:32.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:32.386+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:32.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:32.387+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:32.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:32.627+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:32.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:32.627+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:32.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:32.627+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:32.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:32.628+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:33.417 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:33.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:33.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:33.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:33.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:33.027+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:33.418 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:33.028+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:33.674 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:33.415+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:33.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:33.415+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:33.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:33.415+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:33.675 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:33.416+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:33.672+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:33.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:33.672+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:33.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:33.672+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:33.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:33.672+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:34.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:34.054+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:34.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:34.054+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:34.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:34.054+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:34.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:34.055+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:34.678 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:34.380+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:34.678 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:34.380+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:34.678 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:34.380+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:34.678 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:34.381+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:34.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:34.675+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:34.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:34.675+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:34.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:34.675+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:34.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:34.676+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:35.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:35.093+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:35.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:35.093+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:35.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:35.093+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:35.411 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:35.094+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:35.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:35.408+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:35.693 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:35.408+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:35.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:35.408+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:35.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:35.409+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:35.690+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:35.690+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:35.690+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:35.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:35.691+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:36.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:36.050+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:36.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:36.050+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:36.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:36.050+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:36.441 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:36.052+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:36.438+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:36.438+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:36.438+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:36.440+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:36.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:36.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:36.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:36.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:36.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:36.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:36.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:36.686+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:37.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:37.010+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:37.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:37.010+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:37.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:37.010+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:37.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:37.011+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:37.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:37.363+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23838 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:37.446+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:37.446+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:37.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:37.446+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:37.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:37.446+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:37.969 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:37.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:37.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:37.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:37.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:37.707+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:37.970 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:37.708+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:37.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:37.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:37.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:38.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:37.968+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:38.456+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:38.456+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:38.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:38.456+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:38.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:38.457+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:38.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:38.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:38.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:38.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:38.665+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:39.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:39.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:39.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:39.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:39.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:39.451 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:39.001+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:39.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:39.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:39.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:39.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:39.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:39.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:39.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:39.450+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:39.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:39.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:39.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:39.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:39.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:39.688+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:39.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:39.689+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:40.033+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:40.033+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:40.033+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:40.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:40.033+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:40.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:40.477+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:40.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:40.477+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:40.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:40.477+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:40.952 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:40.477+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:40.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:40.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:40.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:40.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:40.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:40.649+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:40.953 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:40.650+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:41.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:41.047+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:41.359 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:41.047+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:41.359 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:41.047+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:41.359 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:41.048+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:41.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:41.489+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:41.682 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:41.489+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:41.682 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:41.489+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:41.682 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:41.491+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:41.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:41.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:41.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:41.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:41.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:41.680+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:42.369 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:42.078+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:42.369 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:42.079+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:42.369 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:42.079+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:42.369 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:42.080+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:42.642 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:42.366+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 23928 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:42.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:42.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:42.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:42.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:42.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:42.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:42.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:42.528+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:42.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:42.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:42.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:42.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:42.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:42.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:42.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:42.640+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:43.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:43.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:43.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:43.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:43.068+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:43.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:43.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:43.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:43.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:43.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:43.528+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:43.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:43.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:43.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:43.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:43.678+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:44.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:44.044+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:44.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:44.044+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:44.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:44.044+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:44.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:44.046+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:44.683 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:44.511+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:44.684 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:44.511+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:44.684 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:44.511+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:44.684 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:44.512+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:44.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:44.681+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:44.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:44.681+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:44.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:44.681+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:44.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:44.682+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:45.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:45.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:45.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:45.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:45.032+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:45.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:45.506+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:45.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:45.506+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:45.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:45.506+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:45.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:45.507+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:45.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:45.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:45.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:45.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:45.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:45.696+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:45.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:45.697+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:46.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:46.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:46.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:46.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:46.012+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:46.497+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:46.497+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:46.497+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:46.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:46.498+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:46.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:46.660+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:46.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:46.660+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:46.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:46.660+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:46.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:46.660+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:47.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:46.964+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:47.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:46.964+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:47.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:46.964+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:47.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:46.964+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:47.680 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:47.369+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24019 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:47.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:47.470+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:47.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:47.470+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:47.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:47.470+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:47.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:47.470+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:47.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:47.917+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:47.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:47.917+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:47.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:47.917+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:47.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:47.918+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:47.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:47.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:47.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:47.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:47.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:47.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:47.952 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:47.679+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:48.898 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:48.477+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:48.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:48.477+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:48.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:48.477+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:48.899 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:48.478+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:48.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:48.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:48.899 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:48.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:48.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:48.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:48.900 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:48.635+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:48.896+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:48.896+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:48.896+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:49.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:48.896+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:49.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:49.521+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:49.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:49.521+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:49.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:49.521+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:49.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:49.522+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:49.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:49.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:49.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:49.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:49.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:49.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:49.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:49.656+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:50.189 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:49.861+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:50.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:49.861+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:50.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:49.861+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:50.190 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:49.862+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:50.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:50.538+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:50.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:50.538+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:50.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:50.538+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:50.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:50.539+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:50.850 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:50.618+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:50.850 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:50.618+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:50.850 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:50.618+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:50.850 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:50.618+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:51.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:50.846+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:51.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:50.846+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:51.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:50.846+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:51.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:50.847+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:51.888 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:51.565+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:51.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:51.565+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:51.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:51.565+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:51.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:51.566+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:51.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:51.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:51.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:51.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:51.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:51.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:51.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:51.587+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:52.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:51.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:52.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:51.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:52.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:51.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:52.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:51.886+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:52.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:52.371+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24107 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:52.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:52.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:52.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:52.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:52.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:52.527+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:52.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:52.528+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:52.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:52.601+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:52.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:52.601+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:52.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:52.601+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:52.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:52.602+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:53.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:52.917+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:53.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:52.917+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:53.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:52.917+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:53.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:52.918+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:53.890 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:53.532+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:53.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:53.532+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:53.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:53.532+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:53.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:53.533+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:53.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:53.597+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:53.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:53.597+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:53.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:53.597+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:53.892 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:53.598+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:53.888+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:53.888+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:53.888+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:54.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:53.889+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:54.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:54.509+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:54.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:54.509+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:54.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:54.509+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:54.918 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:54.510+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:54.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:54.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:54.918 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:54.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:54.919 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:54.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:54.919 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:54.562+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:55.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:54.915+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:55.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:54.915+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:55.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:54.915+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:55.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:54.916+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:55.924 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:55.471+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:55.924 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:55.471+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:55.924 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:55.471+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:55.924 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:55.472+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:55.924 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:55.552+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:55.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:55.552+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:55.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:55.552+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:55.925 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:55.553+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:56.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:55.921+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:56.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:55.921+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:56.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:55.921+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:56.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:55.922+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:56.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:56.460+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:56.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:56.460+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:56.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:56.460+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:56.889 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:56.461+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:56.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:56.557+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:56.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:56.557+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:56.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:56.557+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:56.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:56.558+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:57.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:56.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:57.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:56.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:57.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:56.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:57.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:56.886+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:57.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:13:57.374+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24209 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:13:57.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:57.454+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:57.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:57.454+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:57.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:57.454+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:57.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:57.455+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:57.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:57.538+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:57.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:57.538+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:57.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:57.538+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:57.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:57.539+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:57.908+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:58.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:57.908+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:58.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:57.908+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:58.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:57.908+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:58.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:58.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:58.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:58.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:58.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:58.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:58.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:58.479+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:58.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:58.558+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:58.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:58.558+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:58.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:58.558+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:58.873 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:58.559+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:58.869+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:59.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:58.869+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:59.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:58.869+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:13:59.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:58.870+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:13:59.909 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:59.520+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:59.909 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:59.520+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:59.909 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:59.520+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:13:59.909 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:13:59.521+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:13:59.909 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:59.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:59.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:59.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:59.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:59.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:13:59.910 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:13:59.586+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:00.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:59.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:00.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:59.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:00.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:59.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:00.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:13:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:13:59.905+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:00.902 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:00.493+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:00.902 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:00.493+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:00.902 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:00.493+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:00.903 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:00.494+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:00.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:00.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:00.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:00.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:00.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:00.562+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:00.903 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:00.563+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:01.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:00.899+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:01.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:00.899+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:01.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:00.899+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:01.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:00.900+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:01.863 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:01.490+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:01.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:01.490+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:01.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:01.490+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:01.864 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:01.491+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:01.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:01.559+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:01.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:01.559+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:01.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:01.559+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:01.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:01.560+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:02.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:01.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:02.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:01.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:02.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:01.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:02.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:01.861+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:02.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:02.377+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24298 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:02.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:02.505+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:02.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:02.505+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:02.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:02.505+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:02.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:02.506+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:02.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:02.529+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:02.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:02.529+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:02.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:02.529+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:02.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:02.529+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:03.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:02.879+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:03.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:02.879+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:03.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:02.879+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:03.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:02.879+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:03.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:03.492+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:03.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:03.492+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:03.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:03.492+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:03.932 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:03.492+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:03.932 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:03.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:03.932 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:03.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:03.932 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:03.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:03.933 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:03.539+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:04.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:03.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:04.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:03.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:04.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:03.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:04.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:03.929+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:04.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:04.508+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:04.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:04.508+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:04.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:04.508+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:04.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:04.509+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:04.896 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:04.524+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:04.524+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:04.524+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:04.897 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:04.524+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:05.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:04.893+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:05.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:04.893+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:05.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:04.893+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:05.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:04.894+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:05.934 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:05.501+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:05.934 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:05.501+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:05.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:05.501+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:05.935 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:05.502+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:05.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:05.505+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:05.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:05.505+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:05.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:05.505+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:05.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:05.506+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:06.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:05.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:06.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:05.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:06.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:05.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:06.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:05.931+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:06.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:06.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:06.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:06.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:06.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:06.548+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:06.470+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:06.470+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:06.470+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:06.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:06.470+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:07.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:06.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:07.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:06.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:07.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:06.956+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:07.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:06.956+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:07.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:07.378+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24390 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:07.448+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:07.448+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:07.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:07.448+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:07.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:07.448+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:07.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:07.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:07.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:07.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:07.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:07.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:07.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:07.570+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:08.421 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:07.982+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:08.422 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:07.982+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:08.422 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:07.982+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:08.422 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:07.982+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:08.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:08.573+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:08.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:08.573+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:08.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:08.573+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:08.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:08.574+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:08.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:08.418+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:08.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:08.418+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:08.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:08.418+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:08.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:08.418+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:09.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:08.978+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:09.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:08.978+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:09.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:08.978+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:09.249 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:08.979+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:09.622 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:09.394+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:09.622 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:09.394+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:09.623 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:09.394+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:09.623 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:09.394+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:09.619+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:09.619+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:09.619+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:09.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:09.620+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:10.414 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:10.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:10.414 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:10.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:10.414 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:10.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:10.414 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:10.007+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:10.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:10.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:10.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:10.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:10.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:10.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:10.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:10.652+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:10.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:10.411+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:10.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:10.411+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:10.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:10.411+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:10.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:10.412+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:11.382 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:10.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:11.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:10.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:11.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:10.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:11.383 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:10.984+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:11.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:11.380+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:11.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:11.380+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:11.648 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:11.380+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:11.649 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:11.380+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:11.644+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:11.644+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:11.644+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:11.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:11.644+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:12.384 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:11.969+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:12.385 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:11.969+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:12.385 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:11.969+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:12.385 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:11.970+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:12.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:12.382+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24479 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:12.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:12.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:12.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:12.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:12.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:12.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:12.596+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:12.391+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:12.391+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:12.391+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:12.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:12.391+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:12.940+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:12.940+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:12.940+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:13.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:12.941+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:13.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:13.554+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:13.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:13.554+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:13.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:13.554+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:13.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:13.556+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:13.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:13.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:13.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:13.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:13.393+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:14.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:13.907+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:14.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:13.907+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:14.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:13.907+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:14.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:13.907+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:14.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:14.543+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:14.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:14.543+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:14.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:14.543+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:14.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:14.545+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:14.360+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:14.360+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:14.360+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:14.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:14.360+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:15.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:14.918+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:15.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:14.918+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:15.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:14.918+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:15.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:14.919+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:15.400+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:15.400+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:15.400+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:15.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:15.400+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:15.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:15.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:15.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:15.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:15.594+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:16.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:15.872+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:16.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:15.872+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:16.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:15.872+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:16.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:15.873+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:16.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:16.370+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:16.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:16.370+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:16.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:16.370+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:16.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:16.371+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:16.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:16.588+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:16.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:16.588+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:16.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:16.588+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:16.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:16.588+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:17.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:16.894+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:17.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:16.894+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:17.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:16.894+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:17.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:16.894+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:17.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:17.385+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24572 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:17.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:17.338+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:17.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:17.338+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:17.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:17.338+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:17.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:17.339+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:17.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:17.550+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:17.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:17.550+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:17.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:17.550+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:17.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:17.551+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:18.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:17.888+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:18.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:17.888+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:18.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:17.888+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:18.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:17.889+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:18.297+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:18.297+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:18.297+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:18.588 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:18.297+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:18.863 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:18.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:18.863 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:18.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:18.863 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:18.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:18.863 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:18.587+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:18.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:18.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:18.860+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:19.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:18.862+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:19.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:19.332+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:19.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:19.332+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:19.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:19.332+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:19.620 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:19.333+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:19.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:19.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:19.874 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:19.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:19.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:19.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:19.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:19.619+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:20.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:19.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:20.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:19.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:20.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:19.871+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:20.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:19.872+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:20.329+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:20.329+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:20.329+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:20.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:20.330+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:20.922 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:20.628+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:20.922 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:20.628+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:20.922 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:20.628+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:20.922 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:20.629+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:21.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:20.919+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:21.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:20.919+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:21.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:20.919+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:21.202 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:20.921+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:21.611 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:21.365+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:21.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:21.365+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:21.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:21.365+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:21.612 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:21.365+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:21.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:21.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:21.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:21.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:21.610+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:22.356 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:21.957+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:22.357 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:21.957+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:22.357 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:21.957+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:22.357 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:21.958+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:22.702 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:22.387+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24661 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:22.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:22.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:22.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:22.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:22.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:22.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:22.703 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:22.586+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:22.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:22.354+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:22.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:22.354+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:22.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:22.354+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:22.703 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:22.355+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:23.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:22.963+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:23.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:22.963+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:23.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:22.963+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:23.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:22.964+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:23.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:23.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:23.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:23.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:23.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:23.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:23.626 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:23.554+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:23.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:23.344+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:23.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:23.344+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:23.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:23.344+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:23.627 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:23.344+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:24.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:23.966+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:24.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:23.966+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:24.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:23.966+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:24.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:23.968+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:24.391+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:24.391+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:24.391+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:24.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:24.392+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:24.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:24.551+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:24.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:24.551+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:24.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:24.551+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:24.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:24.552+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:25.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:24.964+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:25.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:24.964+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:25.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:24.964+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:25.368 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:24.965+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:25.368 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:25.365+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:25.368 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:25.365+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:25.368 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:25.365+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:25.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:25.365+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:25.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:25.562+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:25.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:25.562+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:25.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:25.562+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:25.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:25.563+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:26.353 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:25.991+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:26.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:25.991+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:26.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:25.991+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:26.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:25.992+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:26.610 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:26.351+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:26.610 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:26.351+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:26.610 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:26.351+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:26.610 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:26.352+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:26.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:26.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:26.608+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:26.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:26.609+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:27.392 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:26.958+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:27.392 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:26.958+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:27.393 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:26.958+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:27.393 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:26.960+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:27.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:27.390+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24753 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:27.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:27.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:27.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:27.595+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:27.393+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:27.394+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:27.394+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:27.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:27.394+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:27.913+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:27.913+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:27.913+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:28.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:27.914+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:28.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:28.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:28.569+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:28.570+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:28.349+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:28.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:28.349+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:28.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:28.349+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:28.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:28.350+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:28.905+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:28.905+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:28.905+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:29.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:28.906+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:29.616 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:29.389+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:29.616 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:29.389+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:29.617 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:29.389+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:29.617 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:29.390+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:29.880 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:29.613+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:29.880 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:29.613+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:29.880 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:29.613+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:29.880 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:29.614+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:30.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:29.877+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:30.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:29.877+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:30.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:29.877+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:30.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:29.878+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:30.449 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:30.343+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:30.343+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:30.343+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:30.450 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:30.344+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:30.876 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:30.581+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:30.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:30.581+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:30.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:30.581+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:30.877 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:30.581+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:30.874+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:30.874+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:31.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:30.874+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:31.201 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:30.875+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:31.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:31.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:31.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:31.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:31.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:31.392+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:31.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:31.393+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:31.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:31.606+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:31.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:31.606+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:31.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:31.606+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:31.702 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:31.607+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:32.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:31.884+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:32.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:31.884+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:32.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:31.884+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:32.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:31.885+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:32.700 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:32.392+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24842 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:32.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:32.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:32.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:32.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:32.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:32.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:32.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:32.593+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:32.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:32.367+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:32.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:32.367+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:32.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:32.367+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:32.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:32.367+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:33.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:32.890+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:33.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:32.890+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:33.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:32.890+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:33.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:32.891+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:33.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:33.377+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:33.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:33.377+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:33.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:33.377+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:33.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:33.378+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:33.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:33.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:33.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:33.585+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:33.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:33.586+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:33.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:33.586+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:34.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:33.920+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:34.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:33.920+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:34.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:33.920+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:34.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:33.921+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:34.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:34.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:34.387+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:34.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:34.387+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:34.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:34.619+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:34.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:34.619+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:34.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:34.619+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:34.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:34.619+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:35.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:34.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:35.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:34.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:35.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:34.880+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:35.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:34.881+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:35.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:35.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:35.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:35.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:35.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:35.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:35.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:35.618+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:35.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:35.422+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:35.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:35.422+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:35.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:35.422+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:35.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:35.422+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:36.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:35.864+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:36.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:35.864+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:36.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:35.864+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:36.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:35.865+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:36.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:36.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:36.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:36.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:36.595+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:36.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:36.419+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:36.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:36.419+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:36.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:36.419+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:36.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:36.420+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:37.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:36.844+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:37.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:36.844+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:37.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:36.844+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:37.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:36.845+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:37.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:37.395+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 24935 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:37.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:37.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:37.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:37.565+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:37.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:37.372+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:37.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:37.372+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:37.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:37.372+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:37.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:37.373+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:38.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:37.865+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:37.865+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:37.865+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:38.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:37.866+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:38.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:38.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:38.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:38.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:38.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:38.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:38.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:38.596+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:38.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:38.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:38.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:38.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:38.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:38.417+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:38.702 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:38.418+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:38.881+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:38.881+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:38.881+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:39.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:38.882+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:39.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:39.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:39.608 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:39.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:39.609 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:39.452+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:39.609 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:39.453+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:39.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:39.606+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:39.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:39.606+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:39.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:39.606+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:39.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:39.607+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:40.192 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:39.924+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:40.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:39.924+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:40.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:39.924+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:40.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:39.925+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:40.886 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:40.472+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:40.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:40.472+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:40.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:40.472+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:40.887 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:40.473+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:40.887 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:40.624+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:40.887 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:40.624+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:40.887 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:40.624+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:40.888 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:40.625+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:41.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:40.884+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:41.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:40.884+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:41.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:40.884+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:41.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:40.885+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:41.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:41.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:41.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:41.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:41.723 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:41.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:41.724 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:41.661+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:41.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:41.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:41.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:41.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:41.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:41.458+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:41.724 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:41.459+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:42.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:41.891+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:42.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:41.891+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:42.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:41.891+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:42.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:41.891+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:42.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:42.398+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25024 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:42.496+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:42.496+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:42.496+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:42.497+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:42.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:42.642+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:42.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:42.642+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:42.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:42.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:42.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:42.643+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:43.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:42.916+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:43.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:42.916+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:43.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:42.916+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:43.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:42.917+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:43.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:43.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:43.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:43.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:43.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:43.609+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:43.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:43.609+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:43.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:43.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:43.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:43.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:43.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:43.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:43.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:43.535+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:44.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:43.924+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:44.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:43.924+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:44.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:43.924+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:44.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:43.925+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:44.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:44.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:44.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:44.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:44.942 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:44.618+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:44.943 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:44.619+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:44.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:44.558+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:44.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:44.558+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:44.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:44.558+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:44.943 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:44.559+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:45.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:44.940+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:45.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:44.940+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:44.940+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:45.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:44.941+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:45.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:45.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:45.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:45.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:45.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:45.592+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:45.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:45.593+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:45.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:45.521+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:45.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:45.521+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:45.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:45.521+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:45.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:45.521+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:46.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:45.942+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:45.942+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:45.942+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:46.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:45.943+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:46.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:46.614+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:46.925 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:46.614+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:46.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:46.614+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:46.926 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:46.615+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:46.926 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:46.530+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:46.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:46.530+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:46.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:46.530+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:46.927 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:46.531+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:46.922+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:46.922+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:46.922+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:47.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:46.923+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:47.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:47.401+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25116 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:47.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:47.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:47.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:47.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:47.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:47.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:47.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:47.575+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:47.552+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:47.552+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:47.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:47.552+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:47.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:47.552+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:48.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:47.971+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:47.971+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:47.971+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:48.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:47.972+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:48.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:48.600+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:48.600+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:48.600+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:48.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:48.601+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:48.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:48.571+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:48.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:48.571+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:48.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:48.571+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:48.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:48.572+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:49.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:48.961+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:49.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:48.961+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:49.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:48.961+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:49.248 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:48.962+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:49.938 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:49.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:49.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:49.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:49.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:49.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:49.939 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:49.568+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:49.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:49.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:49.939 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:49.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:49.940 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:49.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:49.941 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:49.535+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:50.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:49.936+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:50.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:49.936+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:50.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:49.936+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:50.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:49.938+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:50.930 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:50.537+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:50.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:50.537+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:50.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:50.537+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:50.931 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:50.538+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:50.931 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:50.582+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:50.931 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:50.582+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:50.931 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:50.582+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:50.931 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:50.583+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:51.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:50.929+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:51.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:50.929+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:51.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:50.929+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:51.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:50.929+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:51.928 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:51.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:51.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:51.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:51.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:51.596+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:51.929 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:51.597+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:51.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:51.587+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:51.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:51.587+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:51.929 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:51.587+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:51.930 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:51.587+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:52.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:51.927+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:52.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:51.927+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:52.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:51.927+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:52.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:51.928+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:52.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:52.404+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25205 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:52.575+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:52.575+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:52.575+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:52.576+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:52.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:52.611+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:52.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:52.611+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:52.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:52.611+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:52.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:52.612+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:53.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:52.897+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:53.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:52.897+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:53.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:52.897+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:53.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:52.898+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:53.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:53.633+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:53.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:53.633+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:53.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:53.633+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:53.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:53.634+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:53.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:53.566+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:53.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:53.566+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:53.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:53.566+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:53.913 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:53.566+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:54.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:53.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:54.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:53.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:54.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:53.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:54.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:53.912+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:54.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:54.653+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:54.871 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:54.653+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:54.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:54.653+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:54.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:54.654+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:54.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:54.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:54.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:54.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:54.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:54.534+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:54.872 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:54.534+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:55.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:54.870+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:55.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:54.870+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:55.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:54.870+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:55.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:54.870+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:55.825 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:55.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:55.825 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:55.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:55.825 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:55.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:55.825 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:55.521+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:55.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:55.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:55.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:55.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:55.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:55.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:55.826 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:55.642+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:56.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:55.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:56.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:55.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:56.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:55.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:56.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:55.824+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:56.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:56.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:56.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:56.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:56.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:56.683+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:56.813 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:56.684+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:56.813 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:56.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:56.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:56.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:56.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:56.520+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:56.814 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:56.520+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:57.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:56.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:57.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:56.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:57.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:56.811+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:57.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:56.812+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:57.663 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:14:57.406+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25298 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:14:57.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:57.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:57.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:57.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:57.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:57.491+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:57.663 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:57.492+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:57.839+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:57.839+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:57.839+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:57.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:57.840+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:57.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:57.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:57.661+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:57.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:57.662+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:58.708 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:58.450+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:58.708 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:58.450+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:58.708 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:58.450+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:58.708 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:58.450+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:59.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:58.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:59.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:58.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:59.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:58.706+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:59.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:58.707+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:58.807+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:58.807+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:58.807+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:59.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:58.808+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:14:59.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:59.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:59.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:59.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:59.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:59.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:14:59.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:14:59.426+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:59.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:59.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:59.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:59.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:59.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:59.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:14:59.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:14:59.684+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:14:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:59.837+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:59.837+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:59.837+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:14:59.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:14:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:14:59.838+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:00.861 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:00.638+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:00.861 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:00.638+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:00.861 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:00.638+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:00.861 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:00.639+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:00.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:00.454+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:00.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:00.454+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:00.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:00.454+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:00.862 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:00.454+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:01.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:00.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:01.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:00.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:01.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:00.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:01.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:00.860+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:01.848 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:01.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:01.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:01.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:01.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:01.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:01.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:01.684+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:01.849 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:01.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:01.849 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:01.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:01.850 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:01.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:01.850 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:01.483+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:02.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:01.847+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:02.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:01.847+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:02.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:01.847+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:02.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:01.848+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:02.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:02.409+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25387 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:02.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:02.437+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:02.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:02.437+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:02.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:02.437+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:02.701 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:02.438+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:03.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:02.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:03.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:02.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:03.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:02.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:03.199 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:02.716+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:03.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:02.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:03.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:02.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:03.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:02.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:03.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:02.856+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:03.832 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:03.475+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:03.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:03.475+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:03.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:03.475+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:03.833 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:03.475+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:03.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:03.714+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:03.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:03.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:03.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:03.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:03.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:03.715+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:04.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:03.831+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:04.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:03.831+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:04.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:03.831+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:04.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:03.832+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:04.687 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:04.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:04.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:04.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:04.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:04.494+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:04.688 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:04.495+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:04.827+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:04.827+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:04.827+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:04.828+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:04.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:04.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:04.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:04.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:04.687+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:05.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:05.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:05.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:05.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:05.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:05.710+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:05.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:05.710+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:05.795 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:05.528+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:05.795 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:05.528+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:05.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:05.528+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:05.796 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:05.529+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:06.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:05.793+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:06.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:05.793+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:06.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:05.793+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:06.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:05.794+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:06.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:06.693+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:06.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:06.693+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:06.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:06.693+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:06.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:06.694+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:06.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:06.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:06.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:06.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:06.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:06.504+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:06.766 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:06.504+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:07.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:06.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:07.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:06.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:07.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:06.763+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:07.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:06.764+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:07.667 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:07.412+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25479 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:07.667 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:07.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:07.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:07.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:07.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:07.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:07.668 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:07.533+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:07.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:07.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:07.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:07.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:07.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:07.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:07.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:07.667+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:07.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:07.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:07.800+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:07.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:07.800+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:08.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:08.694+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:08.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:08.694+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:08.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:08.694+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:08.825 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:08.695+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:08.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:08.525+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:08.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:08.525+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:08.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:08.525+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:08.826 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:08.526+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:09.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:08.823+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:09.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:08.824+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:09.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:08.824+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:09.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:08.824+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:09.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:09.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:09.485+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:09.485+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:10.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:09.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:10.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:09.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:10.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:09.727+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:10.193 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:09.728+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:10.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:09.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:10.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:09.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:10.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:09.778+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:10.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:09.778+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:10.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:10.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:10.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:10.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:10.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:10.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:10.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:10.512+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:11.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:10.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:11.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:10.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:11.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:10.741+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:11.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:10.742+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:10.736+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:10.736+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:10.736+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:11.200 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:10.736+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:11.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:11.479+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:11.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:11.479+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:11.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:11.479+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:11.741 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:11.480+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:11.742 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:11.713+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:11.742 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:11.713+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:11.742 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:11.713+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:11.742 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:11.714+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:12.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:11.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:12.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:11.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:12.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:11.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:12.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:11.740+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:12.670 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:12.415+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25568 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:12.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:12.461+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:12.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:12.461+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:12.670 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:12.461+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:12.671 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:12.462+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:12.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:12.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:12.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:12.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:12.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:12.779+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:12.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:12.780+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:12.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:12.669+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:12.669+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:12.669+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:12.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:12.670+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:13.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:13.620+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:13.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:13.621+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:13.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:13.621+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:13.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:13.621+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:13.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:13.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:13.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:13.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:13.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:13.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:13.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:13.427+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:14.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:13.815+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:14.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:13.815+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:14.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:13.815+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:14.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:13.816+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:14.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:14.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:14.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:14.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:14.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:14.593+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:14.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:14.594+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:14.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:14.474+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:14.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:14.474+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:14.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:14.474+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:14.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:14.474+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:15.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:14.828+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:15.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:14.828+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:15.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:14.828+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:15.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:14.830+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:15.853 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:15.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:15.853 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:15.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:15.854 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:15.589+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:15.854 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:15.590+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:15.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:15.490+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:15.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:15.490+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:15.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:15.490+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:15.854 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:15.491+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:16.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:15.852+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:16.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:15.852+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:16.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:15.852+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:16.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:15.853+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:16.859 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:16.611+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:16.859 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:16.611+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:16.859 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:16.611+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:16.859 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:16.612+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:16.860 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:16.513+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:16.860 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:16.513+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:16.860 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:16.513+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:16.860 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:16.513+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:17.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:16.858+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:17.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:16.858+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:17.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:16.858+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:17.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:16.858+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:17.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:17.418+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25661 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:17.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:17.506+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:17.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:17.506+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:17.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:17.506+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:17.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:17.507+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:17.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:17.630+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:17.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:17.630+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:17.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:17.630+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:17.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:17.630+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:18.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:17.900+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:18.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:17.900+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:18.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:17.900+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:18.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:17.901+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:18.887 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:18.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:18.887 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:18.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:18.888 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:18.622+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:18.888 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:18.623+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:18.888 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:18.525+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:18.888 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:18.525+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:18.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:18.525+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:18.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:18.526+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:19.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:18.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:19.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:18.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:19.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:18.886+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:19.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:18.886+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:19.564+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:19.564+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:19.564+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:19.840 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:19.564+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:19.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:19.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:19.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:19.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:19.841 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:19.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:19.842 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:19.641+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:20.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:19.838+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:20.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:19.838+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:20.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:19.838+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:20.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:19.839+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:20.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:20.612+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:20.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:20.612+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:20.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:20.612+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:20.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:20.612+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:20.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:20.658+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:20.658+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:20.658+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:20.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:20.658+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:20.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:20.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:20.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:20.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:20.951 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:20.850+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:21.844 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:21.581+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:21.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:21.581+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:21.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:21.581+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:21.845 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:21.582+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:21.845 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:21.675+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:21.845 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:21.675+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:21.845 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:21.675+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:21.846 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:21.676+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:22.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:21.843+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:22.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:21.843+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:22.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:21.843+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:22.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:21.844+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:22.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:22.421+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25750 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:22.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:22.647+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:22.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:22.647+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:22.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:22.647+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:22.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:22.647+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:22.554+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:22.554+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:22.554+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:22.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:22.554+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:23.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:22.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:23.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:22.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:23.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:22.862+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:23.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:22.863+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:23.864 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:23.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:23.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:23.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:23.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:23.533+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:23.865 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:23.534+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:23.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:23.600+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:23.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:23.600+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:23.865 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:23.600+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:23.866 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:23.602+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:24.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:23.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:24.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:23.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:24.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:23.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:24.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:23.864+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:24.829 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:24.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:24.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:24.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:24.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:24.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:24.830 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:24.568+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:24.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:24.523+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:24.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:24.523+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:24.830 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:24.523+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:24.831 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:24.523+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:25.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:24.828+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:25.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:24.828+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:25.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:24.828+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:25.200 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:24.830+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:25.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:25.551+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:25.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:25.551+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:25.805 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:25.551+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:25.806 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:25.552+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:25.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:25.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:25.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:25.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:25.809 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:25.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:25.810 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:25.519+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:26.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:25.870+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:26.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:25.870+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:26.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:25.870+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:26.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:25.870+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:26.855 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:26.508+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:26.856 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:26.508+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:26.856 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:26.508+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:26.856 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:26.508+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:26.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:26.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:26.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:26.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:26.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:26.497+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:26.856 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:26.498+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:27.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:26.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:27.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:26.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:27.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:26.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:27.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:26.856+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:27.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:27.423+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25841 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:27.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:27.468+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:27.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:27.468+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:27.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:27.468+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:27.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:27.468+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:27.485+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:27.485+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:27.485+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:27.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:27.485+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:28.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:27.855+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:28.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:27.855+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:28.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:27.855+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:28.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:27.855+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:28.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:28.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:28.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:28.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:28.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:28.426+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:28.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:28.427+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:28.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:28.523+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:28.523+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:28.523+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:28.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:28.524+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:29.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:28.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:29.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:28.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:29.203 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:28.904+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:29.204 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:28.905+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:29.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:29.457+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:29.525 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:29.457+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:29.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:29.457+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:29.526 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:29.457+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:29.920 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:29.524+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:29.921 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:29.524+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:29.921 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:29.524+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:29.921 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:29.524+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:30.193 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:29.916+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:30.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:29.916+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:30.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:29.916+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:30.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:29.917+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:30.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:30.529+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:30.920 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:30.529+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:30.920 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:30.529+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:30.920 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:30.529+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:30.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:30.508+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:30.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:30.508+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:30.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:30.508+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:30.921 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:30.509+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:31.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:30.918+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:31.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:30.918+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:31.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:30.918+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:31.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:30.919+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:31.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:31.499+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:31.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:31.499+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:31.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:31.499+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:31.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:31.500+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:31.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:31.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:31.502+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:31.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:31.502+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:32.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:31.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:32.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:31.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:32.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:31.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:32.428 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:31.950+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:32.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:32.427+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 25930 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:32.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:32.480+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:32.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:32.480+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:32.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:32.480+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:32.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:32.481+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:32.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:32.464+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:32.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:32.464+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:32.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:32.464+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:32.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:32.465+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:33.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:32.962+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:33.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:32.962+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:33.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:32.962+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:33.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:32.963+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:33.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:33.517+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:33.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:33.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:33.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:33.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:33.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:33.518+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:33.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:33.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:33.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:33.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:33.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:33.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:33.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:33.460+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:34.413 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:33.998+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:34.413 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:33.998+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:34.413 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:33.998+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:34.414 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:33.999+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:34.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:34.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:34.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:34.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:34.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:34.482+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:34.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:34.482+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:34.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:34.412+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:34.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:34.412+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:34.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:34.412+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:34.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:34.413+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:35.366 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:35.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:35.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:35.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:35.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:35.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:35.367 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:35.004+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:35.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:35.365+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:35.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:35.365+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:35.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:35.365+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:35.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:35.367+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:35.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:35.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:35.484+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:35.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:35.485+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:36.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:36.004+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:36.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:36.004+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:36.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:36.004+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:36.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:36.005+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:36.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:36.353+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:36.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:36.353+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:36.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:36.353+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:36.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:36.353+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:36.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:36.513+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:36.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:36.513+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:36.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:36.513+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:36.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:36.514+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:37.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:36.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:37.361 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:36.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:37.362 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:36.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:37.362 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:36.985+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:37.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:37.429+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26023 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:37.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:37.539+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:37.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:37.539+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:37.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:37.539+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:37.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:37.540+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:37.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:37.360+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:37.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:37.360+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:37.360+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:37.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:37.362+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:38.324 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:38.016+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:38.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:38.016+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:38.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:38.016+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:38.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:38.017+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:38.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:38.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:38.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:38.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:38.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:38.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:38.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:38.519+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:38.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:38.324+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:38.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:38.324+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:38.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:38.324+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:38.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:38.325+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:39.290 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:39.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:39.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:39.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:39.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:39.030+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:39.291 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:39.031+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:39.548 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:39.290+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:39.548 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:39.290+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:39.548 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:39.290+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:39.549 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:39.291+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:39.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:39.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:39.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:39.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:39.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:39.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:39.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:39.547+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:40.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:40.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:40.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:40.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:40.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:40.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:40.335 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:40.013+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:40.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:40.334+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:40.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:40.334+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:40.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:40.334+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:40.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:40.335+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:40.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:40.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:40.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:40.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:40.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:40.518+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:40.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:40.518+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:41.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:41.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:41.325 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:41.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:41.326 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:41.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:41.326 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:41.001+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:41.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:41.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:41.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:41.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:41.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:41.511+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:41.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:41.511+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:41.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:41.325+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:41.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:41.325+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:41.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:41.325+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:41.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:41.325+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:42.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:41.982+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:42.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:41.982+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:42.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:41.982+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:42.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:41.983+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:42.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:42.432+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26111 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:42.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:42.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:42.535+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:42.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:42.536+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:42.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:42.353+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:42.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:42.353+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:42.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:42.353+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:42.701 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:42.354+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:43.372 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:43.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:43.373 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:43.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:43.373 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:43.003+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:43.373 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:43.003+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:43.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:43.372+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:43.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:43.372+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:43.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:43.372+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:43.628 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:43.373+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:43.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:43.526+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:43.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:43.526+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:43.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:43.526+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:43.629 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:43.526+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:44.237 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:43.973+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:44.237 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:43.973+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:44.237 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:43.973+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:44.237 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:43.974+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:44.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:44.375+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:44.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:44.375+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:44.558 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:44.375+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:44.559 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:44.377+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:44.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:44.557+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:44.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:44.557+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:44.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:44.557+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:44.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:44.558+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:45.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:44.976+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:45.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:44.976+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:45.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:44.976+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:45.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:44.977+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:45.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:45.406+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:45.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:45.406+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:45.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:45.406+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:45.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:45.407+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:45.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:45.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:45.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:45.547+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:45.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:45.548+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:45.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:45.548+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:46.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:46.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:46.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:46.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:46.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:46.023+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:46.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:46.024+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:46.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:46.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:46.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:46.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:46.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:46.550+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:46.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:46.551+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:46.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:46.453+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:46.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:46.453+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:46.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:46.453+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:46.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:46.455+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:47.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:47.044+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:47.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:47.044+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:47.407 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:47.044+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:47.408 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:47.044+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:47.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:47.436+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26203 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:47.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:47.406+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:47.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:47.406+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:47.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:47.406+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:47.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:47.407+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:47.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:47.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:47.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:47.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:47.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:47.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:47.593+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:48.400 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:48.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:48.400 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:48.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:48.401 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:48.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:48.401 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:48.001+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:48.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:48.399+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:48.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:48.399+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:48.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:48.399+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:48.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:48.401+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:48.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:48.595+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:48.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:48.595+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:48.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:48.595+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:48.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:48.596+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:49.246 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:48.973+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:49.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:48.973+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:49.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:48.973+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:49.247 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:48.974+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:49.584 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:49.386+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:49.584 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:49.386+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:49.584 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:49.386+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:49.584 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:49.387+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:49.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:49.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:49.583+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:49.584+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:50.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:50.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:50.358 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:50.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:50.359 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:50.006+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:50.359 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:50.007+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:50.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:50.358+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:50.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:50.358+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:50.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:50.358+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:50.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:50.359+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:50.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:50.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:50.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:50.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:50.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:50.586+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:50.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:50.586+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:51.397 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:50.979+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:51.397 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:50.979+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:51.397 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:50.979+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:51.397 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:50.980+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:51.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:51.396+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:51.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:51.396+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:51.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:51.396+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:51.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:51.397+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:51.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:51.555+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:51.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:51.555+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:51.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:51.555+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:51.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:51.555+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:52.431 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:51.995+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:52.432 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:51.995+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:52.432 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:51.995+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:52.432 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:51.997+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:52.699 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:52.438+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26292 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:52.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:52.431+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:52.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:52.431+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:52.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:52.431+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:52.700 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:52.432+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:52.597+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:52.597+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:52.597+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:52.700 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:52.597+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:53.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:53.032+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:53.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:53.032+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:53.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:53.032+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:53.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:53.033+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:53.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:53.582+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:53.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:53.582+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:53.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:53.582+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:53.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:53.583+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:53.462+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:53.462+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:53.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:53.462+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:53.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:53.463+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:54.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:54.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:54.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:54.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:54.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:54.067+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:54.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:54.068+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:54.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:54.482+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:54.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:54.482+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:54.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:54.482+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:54.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:54.483+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:54.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:54.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:54.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:54.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:54.622+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:55.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:55.090+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:55.090+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:55.090+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:55.450 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:55.092+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:55.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:55.517+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:55.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:55.517+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:55.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:55.517+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:55.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:55.518+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:55.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:55.660+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:55.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:55.660+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:55.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:55.660+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:55.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:55.661+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:56.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:56.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:56.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:56.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:56.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:56.057+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:56.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:56.058+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:56.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:56.542+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:56.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:56.542+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:56.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:56.542+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:56.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:56.543+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:56.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:56.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:56.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:56.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:56.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:56.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:56.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:56.680+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:57.442 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:57.059+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:57.442 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:57.059+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:57.443 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:57.059+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:57.443 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:57.059+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:57.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:15:57.442+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26385 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:15:57.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:57.652+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:57.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:57.652+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:57.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:57.652+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:57.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:57.652+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:57.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:57.498+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:57.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:57.498+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:57.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:57.498+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:57.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:57.499+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:58.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:58.040+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:58.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:58.040+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:58.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:58.040+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:58.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:58.041+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:58.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:58.700+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:58.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:58.700+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:58.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:58.700+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:58.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:58.701+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:58.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:58.504+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:58.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:58.504+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:58.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:58.504+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:58.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:58.504+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:59.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:59.001+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:59.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:59.001+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:59.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:59.001+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:15:59.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:15:59.002+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:15:59.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:59.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:59.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:59.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:59.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:59.711+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:15:59.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:15:59.711+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:15:59.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:59.524+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:59.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:59.524+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:59.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:59.524+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:15:59.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:15:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:15:59.526+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:00.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:00.046+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:00.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:00.046+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:00.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:00.046+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:00.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:00.047+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:00.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:00.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:00.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:00.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:00.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:00.664+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:00.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:00.665+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:00.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:00.517+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:00.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:00.517+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:00.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:00.517+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:00.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:00.518+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:01.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:01.094+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:01.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:01.094+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:01.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:01.094+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:01.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:01.094+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:01.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:01.635+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:01.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:01.635+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:01.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:01.635+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:01.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:01.636+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:01.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:01.483+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:01.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:01.483+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:01.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:01.483+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:01.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:01.484+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:02.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:02.053+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:02.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:02.053+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:02.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:02.053+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:02.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:02.053+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:02.698 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:02.444+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26474 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:02.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:02.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:02.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:02.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:02.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:02.668+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:02.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:02.669+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:02.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:02.494+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:02.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:02.494+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:02.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:02.494+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:02.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:02.495+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:03.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:03.076+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:03.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:03.076+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:03.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:03.076+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:03.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:03.077+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:03.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:03.481+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:03.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:03.481+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:03.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:03.481+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:03.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:03.482+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:03.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:03.717+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:03.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:03.717+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:03.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:03.717+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:03.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:03.718+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:04.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:04.084+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:04.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:04.084+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:04.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:04.084+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:04.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:04.085+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:04.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:04.472+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:04.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:04.472+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:04.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:04.472+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:04.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:04.473+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:04.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:04.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:04.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:04.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:04.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:04.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:04.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:04.706+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:05.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:05.102+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:05.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:05.102+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:05.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:05.102+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:05.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:05.102+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:05.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:05.492+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:05.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:05.492+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:05.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:05.492+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:05.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:05.493+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:05.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:05.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:05.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:05.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:05.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:05.706+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:05.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:05.707+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:06.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:06.134+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:06.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:06.134+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:06.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:06.134+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:06.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:06.135+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:06.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:06.529+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:06.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:06.529+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:06.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:06.529+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:06.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:06.530+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:06.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:06.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:06.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:06.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:06.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:06.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:06.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:06.758+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:07.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:07.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:07.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:07.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:07.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:07.160+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:07.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:07.160+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:07.754 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:07.447+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26566 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:07.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:07.493+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:07.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:07.493+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:07.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:07.493+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:07.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:07.493+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:08.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:07.754+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:08.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:07.754+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:08.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:07.754+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:08.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:07.755+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:08.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:08.144+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:08.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:08.144+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:08.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:08.144+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:08.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:08.145+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:08.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:08.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:08.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:08.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:08.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:08.478+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:08.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:08.478+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:08.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:08.725+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:08.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:08.725+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:08.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:08.725+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:08.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:08.726+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:09.438 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:09.178+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:09.438 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:09.178+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:09.438 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:09.178+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:09.438 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:09.179+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:09.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:09.437+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:09.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:09.437+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:09.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:09.437+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:09.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:09.439+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:09.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:09.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:09.685+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:09.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:09.686+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:10.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:10.180+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:10.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:10.180+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:10.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:10.180+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:10.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:10.181+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:10.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:10.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:10.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:10.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:10.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:10.449+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:10.710 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:10.450+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:11.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:10.710+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:11.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:10.710+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:11.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:10.710+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:11.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:10.710+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:11.498 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:11.225+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:11.498 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:11.225+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:11.498 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:11.225+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:11.499 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:11.225+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:11.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:11.498+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:11.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:11.498+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:11.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:11.498+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:11.755 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:11.499+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:12.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:11.753+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:12.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:11.753+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:12.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:11.753+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:12.187 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:11.754+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:12.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:12.187+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:12.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:12.187+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:12.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:12.187+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:12.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:12.188+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:12.719 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:12.451+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26654 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:12.719 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:12.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:12.719 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:12.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:12.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:12.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:12.720 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:12.452+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:13.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:12.718+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:13.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:12.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:13.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:12.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:13.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:12.719+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:13.629 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:13.231+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:13.629 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:13.231+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:13.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:13.231+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:13.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:13.232+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:13.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:13.422+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:13.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:13.422+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:13.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:13.422+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:13.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:13.423+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:13.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:13.692+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:13.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:13.692+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:13.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:13.692+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:13.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:13.693+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:14.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:14.409+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:14.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:14.409+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:14.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:14.409+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:14.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:14.409+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:14.211+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:14.211+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:14.211+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:14.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:14.212+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:15.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:14.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:15.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:14.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:15.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:14.708+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:15.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:14.709+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:15.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:15.261+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:15.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:15.261+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:15.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:15.261+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:15.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:15.262+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:15.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:15.390+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:15.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:15.390+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:15.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:15.390+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:15.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:15.390+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:16.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:15.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:16.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:15.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:16.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:15.739+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:16.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:15.739+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:16.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:16.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:16.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:16.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:16.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:16.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:16.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:16.390+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:16.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:16.277+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:16.277+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:16.277+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:16.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:16.278+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:17.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:16.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:17.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:16.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:17.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:16.701+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:17.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:16.701+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:17.582 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:17.454+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26747 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:17.583 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:17.381+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:17.583 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:17.381+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:17.583 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:17.381+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:17.583 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:17.382+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:17.583 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:17.297+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:17.583 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:17.297+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:17.584 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:17.297+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:17.584 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:17.298+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:17.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:17.749+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:17.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:17.749+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:17.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:17.749+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:17.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:17.750+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:18.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:18.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:18.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:18.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:18.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:18.389+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:18.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:18.390+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:18.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:18.281+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:18.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:18.281+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:18.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:18.281+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:18.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:18.282+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:19.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:18.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:19.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:18.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:19.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:18.728+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:19.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:18.729+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:19.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:19.378+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:19.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:19.378+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:19.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:19.378+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:19.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:19.379+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:19.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:19.239+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:19.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:19.239+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:19.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:19.239+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:19.700 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:19.239+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:20.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:19.723+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:20.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:19.723+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:20.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:19.723+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:20.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:19.723+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:20.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:20.397+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:20.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:20.397+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:20.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:20.397+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:20.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:20.398+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:20.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:20.270+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:20.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:20.270+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:20.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:20.270+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:20.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:20.270+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:20.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:20.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:20.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:20.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:20.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:20.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:20.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:20.681+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:21.415 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:21.223+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:21.415 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:21.223+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:21.415 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:21.223+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:21.415 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:21.224+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:21.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:21.415+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:21.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:21.415+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:21.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:21.415+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:21.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:21.415+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:22.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:21.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:22.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:21.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:22.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:21.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:22.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:21.714+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:22.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:22.377+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:22.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:22.377+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:22.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:22.377+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:22.457 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:22.378+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:22.457 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:22.201+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:22.458 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:22.201+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:22.458 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:22.201+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:22.458 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:22.202+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:22.948 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:22.457+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26836 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:22.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:22.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:22.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:22.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:22.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:22.680+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:22.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:22.681+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:23.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:23.344+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:23.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:23.344+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:23.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:23.344+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:23.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:23.344+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:23.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:23.239+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:23.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:23.239+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:23.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:23.239+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:23.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:23.240+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:23.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:23.686+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:23.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:23.686+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:23.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:23.686+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:23.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:23.687+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:24.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:24.277+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:24.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:24.277+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:24.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:24.277+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:24.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:24.278+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:24.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:24.358+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:24.358+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:24.358+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:24.450 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:24.359+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:25.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:24.736+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:25.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:24.736+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:25.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:24.736+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:25.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:24.737+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:25.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:25.380+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:25.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:25.380+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:25.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:25.380+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:25.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:25.381+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:25.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:25.266+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:25.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:25.266+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:25.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:25.266+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:25.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:25.267+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:26.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:25.702+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:26.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:25.702+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:26.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:25.702+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:26.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:25.702+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:26.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:26.336+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:26.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:26.336+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:26.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:26.336+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:26.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:26.337+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:26.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:26.274+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:26.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:26.274+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:26.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:26.274+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:26.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:26.275+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:27.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:26.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:27.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:26.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:27.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:26.726+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:27.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:26.727+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:27.588 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:27.459+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 26928 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:27.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:27.296+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:27.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:27.296+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:27.588 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:27.296+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:27.589 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:27.297+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:27.377+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:27.377+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:27.377+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:27.589 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:27.379+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:27.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:27.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:27.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:27.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:27.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:27.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:27.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:27.716+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:28.678 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:28.278+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:28.678 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:28.278+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:28.678 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:28.278+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:28.678 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:28.279+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:28.679 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:28.398+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:28.679 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:28.398+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:28.679 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:28.398+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:28.679 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:28.399+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:28.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:28.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:28.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:28.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:28.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:28.678+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:28.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:28.679+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:29.658 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:29.294+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:29.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:29.294+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:29.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:29.294+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:29.794 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:29.295+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:29.794 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:29.349+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:29.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:29.349+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:29.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:29.349+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:29.795 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:29.350+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:29.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:29.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:29.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:29.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:29.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:29.658+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:29.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:29.658+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:30.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:30.370+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:30.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:30.370+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:30.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:30.370+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:30.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:30.371+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:30.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:30.320+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:30.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:30.320+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:30.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:30.320+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:30.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:30.321+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:30.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:30.693+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:30.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:30.693+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:30.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:30.693+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:30.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:30.694+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:31.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:31.351+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:31.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:31.351+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:31.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:31.351+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:31.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:31.352+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:31.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:31.308+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:31.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:31.308+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:31.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:31.308+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:31.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:31.309+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:32.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:31.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:32.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:31.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:32.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:31.703+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:32.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:31.704+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:32.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:32.319+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:32.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:32.319+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:32.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:32.319+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:32.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:32.320+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:32.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:32.345+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:32.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:32.345+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:32.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:32.345+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:32.591 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:32.346+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:32.592 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:32.463+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27017 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:32.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:32.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:32.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:32.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:32.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:32.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:32.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:32.744+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:33.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:33.395+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:33.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:33.395+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:33.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:33.395+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:33.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:33.396+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:33.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:33.337+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:33.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:33.337+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:33.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:33.337+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:33.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:33.338+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:33.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:33.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:33.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:33.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:33.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:33.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:33.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:33.706+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:34.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:34.311+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:34.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:34.311+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:34.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:34.311+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:34.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:34.312+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:34.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:34.440+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:34.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:34.440+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:34.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:34.440+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:34.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:34.441+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:34.734+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:34.734+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:34.734+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:34.735+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:35.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:35.293+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:35.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:35.294+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:35.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:35.294+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:35.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:35.294+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:35.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:35.409+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:35.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:35.409+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:35.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:35.409+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:35.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:35.410+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:36.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:35.747+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:36.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:35.747+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:36.199 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:35.747+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:36.200 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:35.747+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:36.362 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:36.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:36.362 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:36.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:36.364 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:36.361+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:36.364 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:36.361+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:36.364 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:36.325+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:36.365 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:36.325+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:36.365 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:36.325+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:36.365 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:36.326+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:37.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:36.790+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:37.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:36.790+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:37.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:36.790+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:37.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:36.791+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:37.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:37.466+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27106 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:37.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:37.383+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:37.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:37.383+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:37.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:37.383+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:37.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:37.384+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:37.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:37.364+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:37.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:37.364+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:37.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:37.364+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:37.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:37.365+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:38.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:37.753+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:38.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:37.753+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:38.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:37.753+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:38.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:37.753+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:38.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:38.387+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:38.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:38.387+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:38.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:38.387+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:38.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:38.388+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:38.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:38.414+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:38.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:38.414+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:38.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:38.414+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:38.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:38.415+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:39.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:38.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:39.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:38.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:39.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:38.743+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:39.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:38.744+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:39.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:39.366+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:39.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:39.366+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:39.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:39.366+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:39.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:39.367+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:39.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:39.345+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:39.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:39.345+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:39.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:39.345+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:39.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:39.347+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:40.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:39.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:40.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:39.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:40.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:39.713+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:40.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:39.713+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:40.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:40.338+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:40.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:40.338+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:40.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:40.338+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:40.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:40.339+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:40.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:40.388+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:40.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:40.388+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:40.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:40.388+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:40.449 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:40.389+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:40.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:40.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:40.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:40.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:40.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:40.673+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:40.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:40.673+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:41.642 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:41.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:41.642 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:41.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:41.642 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:41.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:41.643 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:41.435+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:41.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:41.376+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:41.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:41.376+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:41.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:41.377+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:41.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:41.377+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:41.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:41.642+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:41.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:41.642+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:41.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:41.642+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:41.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:41.643+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:42.603 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:42.469+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27195 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:42.604 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:42.347+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:42.604 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:42.347+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:42.605 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:42.347+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:42.605 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:42.348+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:42.605 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:42.431+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:42.605 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:42.431+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:42.606 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:42.431+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:42.606 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:42.431+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:42.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:42.604+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:42.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:42.604+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:42.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:42.604+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:42.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:42.604+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:43.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:43.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:43.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:43.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:43.630 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:43.430+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:43.631 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:43.431+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:43.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:43.323+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:43.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:43.323+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:43.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:43.323+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:43.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:43.323+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:43.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:43.630+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:43.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:43.630+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:43.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:43.630+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:43.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:43.631+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:44.619 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:44.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:44.620 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:44.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:44.620 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:44.434+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:44.620 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:44.436+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:44.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:44.278+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:44.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:44.278+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:44.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:44.278+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:44.620 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:44.278+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:44.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:44.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:44.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:44.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:44.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:44.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:44.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:44.620+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:45.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:45.235+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:45.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:45.235+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:45.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:45.235+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:45.627 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:45.235+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:45.627 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:45.412+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:45.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:45.412+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:45.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:45.412+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:45.628 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:45.413+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:45.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:45.627+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:45.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:45.627+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:45.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:45.627+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:45.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:45.627+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:46.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:46.231+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:46.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:46.231+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:46.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:46.231+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:46.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:46.232+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:46.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:46.423+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:46.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:46.423+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:46.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:46.423+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:46.635 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:46.424+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:46.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:46.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:46.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:46.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:46.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:46.634+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:46.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:46.635+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:47.595 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:47.472+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27287 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:47.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:47.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:47.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:47.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:47.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:47.416+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:47.596 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:47.416+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:47.597 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:47.280+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:47.597 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:47.280+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:47.597 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:47.280+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:47.597 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:47.281+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:47.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:47.670+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:47.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:47.670+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:47.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:47.670+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:47.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:47.670+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:48.630 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:48.280+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:48.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:48.280+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:48.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:48.280+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:48.631 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:48.281+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:48.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:48.423+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:48.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:48.423+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:48.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:48.423+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:48.632 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:48.424+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:48.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:48.629+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:48.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:48.629+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:48.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:48.629+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:48.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:48.630+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:49.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:49.253+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:49.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:49.253+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:49.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:49.253+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:49.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:49.254+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:49.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:49.436+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:49.680 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:49.436+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:49.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:49.436+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:49.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:49.437+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:49.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:49.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:49.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:49.679+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:49.680+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:50.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:50.261+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:50.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:50.261+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:50.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:50.261+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:50.449 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:50.261+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:50.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:50.448+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:50.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:50.448+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:50.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:50.448+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:50.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:50.449+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:50.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:50.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:50.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:50.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:50.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:50.639+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:50.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:50.640+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:51.680 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:51.270+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:51.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:51.270+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:51.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:51.270+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:51.681 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:51.271+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:51.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:51.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:51.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:51.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:51.681 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:51.441+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:51.682 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:51.442+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:51.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:51.681+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:51.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:51.681+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:51.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:51.681+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:51.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:51.682+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:52.599 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:52.475+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27376 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:52.599 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:52.491+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:52.600 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:52.491+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:52.600 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:52.491+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:52.600 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:52.491+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:52.600 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:52.286+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:52.600 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:52.286+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:52.600 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:52.286+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:52.600 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:52.287+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:52.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:52.677+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:52.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:52.677+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:52.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:52.677+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:52.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:52.677+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:53.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:53.328+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:53.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:53.328+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:53.632 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:53.328+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:53.633 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:53.329+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:53.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:53.519+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:53.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:53.519+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:53.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:53.519+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:53.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:53.520+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:53.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:53.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:53.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:53.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:53.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:53.655+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:53.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:53.656+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:54.637 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:54.363+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:54.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:54.363+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:54.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:54.363+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:54.638 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:54.364+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:54.638 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:54.485+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:54.638 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:54.485+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:54.638 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:54.485+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:54.639 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:54.486+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:54.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:54.638+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:54.891 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:54.638+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:54.892 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:54.638+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:54.892 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:54.638+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:55.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:55.525+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:55.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:55.525+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:55.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:55.525+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:55.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:55.526+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:55.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:55.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:55.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:55.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:55.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:55.621+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:55.698 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:55.622+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:55.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:55.407+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:55.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:55.407+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:55.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:55.407+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:55.699 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:55.408+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:56.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:56.452+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:56.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:56.452+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:56.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:56.452+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:56.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:56.454+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:56.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:56.643+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:56.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:56.643+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:56.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:56.643+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:56.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:56.643+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:56.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:56.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:56.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:56.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:56.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:56.576+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:56.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:56.576+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:57.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:16:57.478+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27468 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:16:57.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:57.423+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:57.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:57.423+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:57.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:57.423+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:57.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:57.424+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:57.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:57.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:57.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:57.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:57.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:57.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:57.699 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:57.592+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:57.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:57.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:57.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:57.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:57.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:57.620+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:57.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:57.620+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:58.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:58.455+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:58.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:58.455+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:58.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:58.455+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:58.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:58.456+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:58.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:58.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:58.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:58.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:58.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:58.569+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:58.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:58.570+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:16:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:58.636+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:58.636+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:58.636+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:58.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:58.636+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:59.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:59.448+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:59.544 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:59.448+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:59.545 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:59.448+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:16:59.545 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:16:59.448+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:59.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:59.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:59.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:59.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:59.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:59.592+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:16:59.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:16:59.593+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:16:59.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:59.545+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:59.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:59.545+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:59.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:59.545+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:16:59.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:16:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:16:59.546+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:00.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:00.642+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:00.642+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:00.950 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:00.642+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:00.951 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:00.642+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:00.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:00.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:00.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:00.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:00.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:00.450+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:00.951 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:00.451+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:00.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:00.543+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:00.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:00.543+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:00.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:00.543+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:00.952 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:00.544+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:01.683 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:01.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:01.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:01.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:01.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:01.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:01.684 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:01.590+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:01.684 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:01.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:01.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:01.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:01.685 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:01.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:01.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:01.434+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:01.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:01.434+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:01.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:01.434+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:01.685 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:01.435+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:01.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:01.684+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:02.697 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:02.482+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27558 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:02.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:02.541+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:02.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:02.541+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:02.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:02.541+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:02.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:02.542+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:02.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:02.440+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:02.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:02.440+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:02.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:02.440+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:02.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:02.440+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:02.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:02.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:02.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:02.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:02.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:02.683+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:02.699 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:02.684+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:03.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:03.490+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:03.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:03.490+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:03.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:03.490+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:03.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:03.492+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:03.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:03.686+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:03.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:03.686+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:03.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:03.686+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:03.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:03.687+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:03.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:03.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:03.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:03.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:03.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:03.570+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:04.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:04.480+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:04.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:04.480+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:04.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:04.480+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:04.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:04.480+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:04.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:04.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:04.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:04.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:04.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:04.699+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:04.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:04.700+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:04.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:04.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:04.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:04.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:04.614+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:04.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:04.614+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:05.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:05.489+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:05.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:05.489+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:05.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:05.489+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:05.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:05.490+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:05.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:05.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:05.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:05.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:05.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:05.592+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:05.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:05.593+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:05.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:05.723+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:05.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:05.723+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:05.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:05.723+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:05.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:05.723+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:06.762 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:06.489+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:06.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:06.489+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:06.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:06.489+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:06.763 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:06.490+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:06.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:06.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:06.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:06.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:06.763 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:06.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:06.764 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:06.618+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:07.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:06.763+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:07.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:06.763+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:07.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:06.763+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:07.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:06.764+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:07.789 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:07.485+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27649 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:07.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:07.464+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:07.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:07.464+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:07.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:07.464+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:07.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:07.465+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:07.790 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:07.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:07.790 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:07.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:07.790 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:07.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:07.790 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:07.617+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:08.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:07.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:08.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:07.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:08.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:07.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:08.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:07.790+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:08.760 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:08.493+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:08.760 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:08.493+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:08.760 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:08.493+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:08.761 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:08.493+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:08.761 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:08.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:08.761 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:08.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:08.761 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:08.617+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:08.761 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:08.618+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:09.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:08.760+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:09.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:08.760+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:09.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:08.760+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:09.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:08.761+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:09.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:09.502+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:09.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:09.502+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:09.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:09.502+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:09.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:09.503+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:09.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:09.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:09.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:09.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:09.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:09.590+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:09.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:09.591+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:09.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:09.720+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:09.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:09.720+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:09.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:09.720+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:09.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:09.721+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:10.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:10.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:10.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:10.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:10.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:10.737+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:10.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:10.737+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:10.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:10.496+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:10.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:10.496+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:10.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:10.496+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:10.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:10.497+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:10.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:10.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:10.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:10.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:10.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:10.570+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:10.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:10.572+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:11.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:11.534+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:11.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:11.534+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:11.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:11.534+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:11.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:11.535+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:11.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:11.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:11.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:11.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:11.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:11.697+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:11.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:11.698+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:11.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:11.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:11.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:11.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:11.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:11.589+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:11.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:11.589+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:12.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:12.488+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27739 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:12.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:12.586+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:12.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:12.586+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:12.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:12.586+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:12.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:12.587+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:12.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:12.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:12.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:12.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:12.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:12.574+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:12.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:12.574+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:12.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:12.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:12.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:12.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:12.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:12.715+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:12.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:12.716+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:13.633 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:13.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:13.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:13.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:13.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:13.540+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:13.634 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:13.540+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:13.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:13.541+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:13.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:13.541+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:13.634 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:13.541+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:13.635 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:13.542+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:13.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:13.704+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:13.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:13.704+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:13.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:13.704+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:13.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:13.704+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:14.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:14.508+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:14.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:14.508+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:14.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:14.508+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:14.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:14.508+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:14.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:14.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:14.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:14.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:14.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:14.574+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:14.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:14.576+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:14.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:14.744+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:14.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:14.744+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:14.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:14.744+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:14.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:14.744+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:15.545+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:15.545+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:15.545+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:15.546+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:15.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:15.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:15.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:15.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:15.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:15.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:15.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:15.620+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:15.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:15.752+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:15.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:15.752+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:15.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:15.752+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:15.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:15.753+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:16.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:16.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:16.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:16.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:16.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:16.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:16.697 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:16.632+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:16.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:16.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:16.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:16.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:16.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:16.553+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:16.698 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:16.554+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:17.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:16.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:17.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:16.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:17.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:16.757+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:17.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:16.758+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:17.947 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:17.491+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27831 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:17.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:17.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:17.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:17.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:17.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:17.662+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:17.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:17.663+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:17.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:17.561+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:17.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:17.561+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:17.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:17.561+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:17.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:17.561+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:17.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:17.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:17.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:17.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:17.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:17.719+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:17.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:17.720+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:18.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:18.546+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:18.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:18.546+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:18.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:18.546+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:18.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:18.546+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:18.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:18.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:18.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:18.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:18.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:18.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:18.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:18.646+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:18.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:18.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:18.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:18.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:18.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:18.741+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:18.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:18.741+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:19.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:19.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:19.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:19.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:19.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:19.630+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:19.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:19.632+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:19.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:19.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:19.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:19.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:19.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:19.705+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:19.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:19.706+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:19.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:19.544+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:19.544+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:19.544+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:19.950 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:19.546+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:20.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:20.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:20.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:20.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:20.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:20.539+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:20.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:20.540+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:20.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:20.709+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:20.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:20.709+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:20.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:20.709+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:20.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:20.710+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:20.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:20.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:20.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:20.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:20.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:20.600+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:20.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:20.600+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:21.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:21.563+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:21.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:21.563+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:21.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:21.563+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:21.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:21.564+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:21.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:21.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:21.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:21.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:21.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:21.645+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:21.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:21.646+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:21.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:21.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:21.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:21.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:21.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:21.684+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:21.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:21.684+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:22.946 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:22.495+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 27921 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:22.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:22.690+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:22.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:22.690+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:22.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:22.690+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:22.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:22.690+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:22.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:22.528+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:22.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:22.528+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:22.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:22.528+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:22.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:22.529+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:22.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:22.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:22.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:22.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:22.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:22.727+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:22.949 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:22.727+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:23.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:23.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:23.770 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:23.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:23.771 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:23.649+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:23.771 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:23.650+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:23.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:23.499+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:23.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:23.499+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:23.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:23.499+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:23.771 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:23.500+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:24.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:23.771+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:24.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:23.771+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:24.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:23.771+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:24.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:23.772+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:24.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:24.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:24.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:24.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:24.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:24.469+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:24.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:24.470+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:24.766 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:24.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:24.766 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:24.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:24.766 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:24.659+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:24.766 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:24.660+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:25.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:24.766+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:25.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:24.766+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:25.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:24.766+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:25.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:24.767+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:25.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:25.456+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:25.784 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:25.456+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:25.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:25.456+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:25.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:25.457+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:25.785 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:25.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:25.785 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:25.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:25.785 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:25.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:25.785 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:25.673+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:26.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:25.785+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:26.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:25.785+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:26.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:25.785+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:26.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:25.786+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:26.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:26.437+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:26.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:26.437+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:26.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:26.437+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:26.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:26.438+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:26.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:26.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:26.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:26.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:26.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:26.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:26.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:26.790+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:26.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:26.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:26.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:26.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:26.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:26.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:26.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:26.696+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:27.696 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:27.498+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28012 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:27.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:27.446+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:27.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:27.446+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:27.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:27.446+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:27.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:27.447+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:27.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:27.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:27.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:27.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:27.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:27.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:27.698 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:27.688+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:28.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:27.819+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:27.819+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:27.819+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:27.820+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:28.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:28.686+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:28.808 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:28.686+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:28.809 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:28.686+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:28.809 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:28.687+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:28.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:28.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:28.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:28.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:28.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:28.459+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:28.809 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:28.460+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:29.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:28.809+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:29.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:28.809+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:29.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:28.809+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:29.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:28.810+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:29.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:29.447+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:29.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:29.447+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:29.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:29.447+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:29.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:29.448+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:30.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:29.711+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:30.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:29.711+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:30.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:29.711+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:30.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:29.712+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:30.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:29.828+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:30.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:29.828+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:30.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:29.828+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:30.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:29.829+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:30.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:30.407+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:30.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:30.407+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:30.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:30.407+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:30.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:30.408+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:31.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:30.846+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:31.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:30.846+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:31.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:30.846+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:31.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:30.847+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:31.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:30.747+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:31.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:30.747+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:31.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:30.747+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:31.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:30.748+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:31.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:31.393+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:31.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:31.393+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:31.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:31.393+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:31.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:31.394+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:32.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:31.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:32.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:31.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:32.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:31.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:32.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:31.765+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:32.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:31.799+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:32.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:31.799+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:32.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:31.799+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:32.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:31.800+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:32.696 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:32.501+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28102 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:32.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:32.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:32.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:32.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:32.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:32.441+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:32.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:32.441+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:33.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:32.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:33.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:32.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:33.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:32.803+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:33.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:32.804+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:33.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:32.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:33.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:32.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:33.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:32.750+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:33.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:32.751+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:33.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:33.434+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:33.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:33.434+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:33.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:33.434+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:33.697 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:33.436+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:34.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:33.809+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:34.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:33.809+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:34.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:33.809+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:34.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:33.810+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:34.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:33.781+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:34.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:33.781+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:34.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:33.781+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:34.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:33.782+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:34.765 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:34.464+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:34.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:34.464+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:34.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:34.464+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:34.766 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:34.466+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:35.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:34.834+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:35.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:34.834+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:35.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:34.834+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:35.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:34.835+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:34.766+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:34.766+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:34.766+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:34.767+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:35.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:35.491+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:35.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:35.491+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:35.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:35.491+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:35.785 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:35.492+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:36.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:35.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:36.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:35.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:36.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:35.786+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:36.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:35.787+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:36.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:35.801+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:36.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:35.801+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:36.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:35.801+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:36.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:35.802+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:36.788 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:36.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:36.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:36.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:36.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:36.473+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:36.789 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:36.474+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:37.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:36.826+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:37.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:36.826+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:37.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:36.826+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:37.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:36.827+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:37.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:36.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:37.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:36.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:37.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:36.790+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:37.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:36.791+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:37.753 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:37.504+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28194 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:37.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:37.520+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:37.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:37.520+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:37.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:37.520+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:37.754 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:37.522+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:38.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:37.793+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:38.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:37.793+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:38.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:37.793+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:38.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:37.794+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:38.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:37.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:38.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:37.754+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:38.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:37.755+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:38.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:37.756+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:38.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:38.503+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:38.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:38.503+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:38.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:38.503+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:38.812 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:38.504+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:38.813 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:38.710+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:38.813 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:38.710+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:38.813 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:38.710+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:38.813 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:38.712+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:39.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:38.813+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:39.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:38.813+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:39.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:38.813+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:39.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:38.814+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:39.833 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:39.523+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:39.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:39.523+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:39.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:39.523+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:39.834 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:39.524+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:39.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:39.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:39.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:39.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:39.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:39.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:39.834 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:39.696+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:40.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:39.834+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:40.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:39.834+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:40.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:39.834+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:40.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:39.835+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:40.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:40.655+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:40.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:40.655+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:40.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:40.655+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:40.816 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:40.656+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:40.817 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:40.498+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:40.817 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:40.498+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:40.817 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:40.498+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:40.817 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:40.499+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:41.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:40.817+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:41.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:40.817+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:41.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:40.817+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:41.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:40.818+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:41.850 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:41.526+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:41.850 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:41.526+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:41.850 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:41.526+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:41.850 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:41.527+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:41.851 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:41.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:41.851 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:41.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:41.851 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:41.637+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:41.851 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:41.638+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:42.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:41.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:42.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:41.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:42.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:41.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:42.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:41.851+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:42.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:42.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:42.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:42.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:42.896 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:42.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:42.897 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:42.628+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:42.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:42.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:42.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:42.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:42.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:42.548+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:42.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:42.549+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:42.898 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:42.507+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28284 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:43.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:42.897+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:43.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:42.897+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:43.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:42.897+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:43.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:42.898+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:43.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:43.579+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:43.926 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:43.579+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:43.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:43.579+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:43.927 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:43.580+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:43.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:43.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:43.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:43.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:43.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:43.594+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:43.927 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:43.594+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:44.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:43.927+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:44.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:43.927+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:44.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:43.927+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:44.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:43.928+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:44.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:44.616+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:44.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:44.616+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:44.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:44.616+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:44.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:44.616+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:44.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:44.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:44.914 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:44.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:44.914 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:44.559+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:44.914 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:44.560+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:45.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:44.913+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:45.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:44.913+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:45.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:44.913+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:45.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:44.914+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:45.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:45.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:45.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:45.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:45.912 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:45.666+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:45.913 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:45.667+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:45.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:45.594+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:45.913 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:45.594+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:45.914 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:45.594+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:45.914 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:45.594+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:46.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:45.913+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:46.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:45.913+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:46.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:45.913+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:46.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:45.913+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:46.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:46.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:46.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:46.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:46.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:46.596+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:46.906 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:46.596+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:46.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:46.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:46.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:46.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:46.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:46.716+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:46.907 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:46.716+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:47.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:46.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:47.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:46.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:47.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:46.907+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:47.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:46.908+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:47.940 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:47.511+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28373 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:47.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:47.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:47.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:47.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:47.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:47.717+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:47.941 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:47.718+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:47.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:47.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:47.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:47.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:47.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:47.641+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:47.942 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:47.642+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:48.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:47.942+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:48.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:47.942+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:48.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:47.942+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:48.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:47.942+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:48.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:48.671+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:48.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:48.671+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:48.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:48.671+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:48.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:48.672+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:48.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:48.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:48.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:48.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:48.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:48.902+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:48.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:48.902+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:48.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:48.680+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:48.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:48.680+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:48.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:48.680+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:48.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:48.681+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:49.362 ERROR:paramiko.transport:Socket exception: No route to host (113) 2023-12-19T19:17:49.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-19T19:17:49.374 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2023-12-19T19:17:49.375 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-19T19:17:49.395 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-19T19:17:49.408 INFO:tasks.cephadm:Teardown begin 2023-12-19T19:17:49.408 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1794, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-19T19:17:49.415 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-19T19:17:49.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:17:49.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:17:49.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:49.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:49.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:49.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:49.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:49.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:49.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:49.682+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:49.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:49.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:49.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:49.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:49.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:49.672+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:49.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:49.672+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:49.872+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:49.872+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:49.872+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:49.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:49.872+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:50.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:50.836+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:50.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:50.836+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:50.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:50.836+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:50.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:50.837+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:50.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:50.651+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:50.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:50.651+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:50.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:50.651+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:50.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:50.652+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:50.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:50.635+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:50.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:50.635+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:50.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:50.635+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:50.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:50.636+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:51.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:51.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:51.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:51.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:51.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:51.824+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:51.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:51.824+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:51.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:51.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:51.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:51.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:51.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:51.628+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:51.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:51.629+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:51.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:51.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:51.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:51.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:51.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:51.617+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:51.949 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:51.619+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:52.529 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:17:52.848 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:52.514+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28463 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:52.848 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:52.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:52.849 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:52.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:52.849 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:52.619+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:52.849 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:52.621+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:52.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:52.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:52.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:52.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:52.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:52.640+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:52.849 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:52.641+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:53.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:52.849+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:53.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:52.849+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:53.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:52.849+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:53.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:52.850+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:53.874 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:53.637+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:53.875 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:53.637+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:53.875 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:53.637+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:53.875 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:53.638+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:53.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:53.571+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:53.875 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:53.571+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:53.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:53.571+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:53.876 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:53.573+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:54.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:53.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:54.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:53.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:54.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:53.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:54.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:53.876+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:54.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:54.663+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:54.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:54.663+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:54.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:54.663+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:54.919 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:54.664+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:54.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:54.558+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:54.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:54.558+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:54.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:54.558+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:54.920 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:54.559+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:55.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:54.920+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:55.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:54.920+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:55.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:54.920+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:55.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:54.921+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:55.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:55.641+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:55.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:55.642+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:55.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:55.642+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:55.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:55.643+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:55.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:55.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:55.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:55.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:55.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:55.601+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:55.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:55.603+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:56.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:55.965+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:56.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:55.965+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:56.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:55.965+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:56.331 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:55.965+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:56.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:56.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:56.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:56.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:56.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:56.616+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:56.916 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:56.617+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:56.916 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:56.650+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:56.916 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:56.650+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:56.916 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:56.650+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:56.916 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:56.651+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:57.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:56.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:57.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:56.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:57.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:56.916+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:57.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:56.917+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:57.946 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:17:57.518+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28553 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:17:57.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:57.650+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:57.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:57.650+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:57.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:57.650+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:57.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:57.651+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:57.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:57.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:57.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:57.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:57.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:57.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:57.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:57.687+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:58.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:57.966+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:58.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:57.966+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:58.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:57.966+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:58.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:57.966+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:58.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:58.694+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:58.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:58.694+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:58.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:58.694+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:58.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:58.696+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:58.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:58.681+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:58.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:58.681+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:58.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:58.681+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:58.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:58.682+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:59.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:59.004+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:59.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:59.004+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:59.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:59.004+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:17:59.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:59.004+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:17:59.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:59.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:59.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:59.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:59.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:59.646+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:17:59.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:17:59.646+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:17:59.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:59.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:59.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:59.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:59.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:59.686+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:17:59.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:17:59.687+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:00.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:59.993+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:00.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:59.993+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:00.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:59.993+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:00.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:17:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:17:59.993+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:00.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:00.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:00.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:00.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:00.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:00.682+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:00.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:00.683+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:00.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:00.657+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:00.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:00.657+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:00.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:00.657+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:00.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:00.658+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:01.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:00.947+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:01.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:00.947+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:01.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:00.947+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:01.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:00.947+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:01.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:18:01.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:18:01.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:01.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:01.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:01.693+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:01.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:01.694+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:01.909 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:01.695+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:01.910 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:01.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:01.910 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:01.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:01.910 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:01.651+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:01.910 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:01.651+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:02.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:01.910+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:02.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:01.910+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:02.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:01.910+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:02.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:01.911+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:02.739 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:02.521+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28642 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:02.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:02.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:02.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:02.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:02.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:02.643+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:02.740 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:02.643+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:03.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:02.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:03.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:02.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:03.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:02.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:03.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:02.741+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:03.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:02.872+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:03.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:02.872+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:03.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:02.872+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:03.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:02.873+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:03.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:03.623+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:03.891 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:03.623+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:03.892 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:03.623+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:03.892 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:03.624+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:03.892 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:03.751+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:03.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:03.751+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:03.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:03.751+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:03.893 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:03.752+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:04.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:03.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:04.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:03.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:04.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:03.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:04.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:03.893+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:04.593 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:18:04.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:04.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:04.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:04.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:04.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:04.626+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:04.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:04.627+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:04.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:04.724+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:04.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:04.724+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:04.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:04.724+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:04.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:04.725+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:04.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:04.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:04.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:04.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:04.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:04.851+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:04.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:04.852+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:05.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:05.612+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:05.872 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:05.612+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:05.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:05.612+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:05.873 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:05.612+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:05.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:05.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:05.873 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:05.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:05.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:05.687+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:05.874 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:05.688+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:06.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:05.873+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:06.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:05.873+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:06.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:05.873+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:06.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:05.874+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:06.856 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:06.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:06.856 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:06.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:06.857 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:06.567+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:06.857 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:06.568+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:06.857 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:06.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:06.857 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:06.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:06.857 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:06.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:06.857 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:06.696+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:07.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:06.858+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:07.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:06.858+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:07.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:06.858+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:07.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:06.858+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:07.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:07.544+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:07.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:07.544+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:07.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:07.544+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:07.824 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:07.545+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:07.824 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:07.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:07.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:07.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:07.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:07.701+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:07.825 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:07.701+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:07.825 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:07.525+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28733 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:08.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:07.825+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:08.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:07.825+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:08.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:07.825+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:08.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:07.826+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:08.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:08.513+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:08.786 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:08.513+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:08.787 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:08.513+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:08.787 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:08.514+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:08.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:08.706+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:08.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:08.706+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:08.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:08.706+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:08.787 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:08.708+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:09.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:08.788+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:09.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:08.788+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:09.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:08.788+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:09.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:08.788+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:09.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:09.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:09.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:09.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:09.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:09.789+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:09.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:09.790+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:09.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:09.561+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:09.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:09.561+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:09.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:09.561+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:09.948 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:09.561+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:09.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:09.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:09.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:09.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:09.949 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:09.695+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:09.950 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:09.695+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:10.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:10.806+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:10.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:10.806+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:10.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:10.806+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:10.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:10.806+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:10.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:10.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:10.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:10.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:10.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:10.564+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:10.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:10.565+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:10.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:10.690+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:10.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:10.690+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:10.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:10.690+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:10.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:10.690+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:11.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:11.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:11.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:11.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:11.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:11.830+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:11.946 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:11.830+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:11.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:11.599+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:11.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:11.599+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:11.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:11.599+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:11.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:11.599+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:11.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:11.724+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:11.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:11.724+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:11.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:11.724+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:11.948 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:11.725+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:12.802 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:12.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:12.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:12.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:12.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:12.692+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:12.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:12.692+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:12.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:12.624+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:12.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:12.624+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:12.803 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:12.624+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:12.804 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:12.625+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:12.804 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:12.528+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28822 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:13.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:12.847+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:13.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:12.847+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:13.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:12.847+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:13.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:12.848+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:13.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:13.673+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:13.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:13.673+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:13.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:13.673+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:13.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:13.674+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:13.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:13.714+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:13.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:13.714+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:13.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:13.714+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:13.947 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:13.715+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:13.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:13.845+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:13.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:13.845+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:13.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:13.845+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:13.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:13.846+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:14.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:14.686+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:14.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:14.686+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:14.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:14.686+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:14.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:14.687+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:14.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:14.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:14.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:14.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:14.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:14.695+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:14.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:14.696+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:14.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:14.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:14.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:14.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:14.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:14.860+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:14.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:14.861+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:15.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:15.731+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:15.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:15.731+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:15.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:15.731+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:15.946 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:15.732+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:15.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:15.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:15.660+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:15.947 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:15.661+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:15.947 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:15.904+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:15.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:15.904+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:15.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:15.904+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:15.948 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:15.904+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:16.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:18:16.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:18:16.914 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:16.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:16.914 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:16.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:16.914 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:16.665+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:16.914 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:16.666+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:16.914 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:16.706+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:16.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:16.706+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:16.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:16.706+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:16.915 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:16.707+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:16.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:16.914+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:16.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:16.915+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:16.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:16.915+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:16.915 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:16.915+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:17.934 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:17.531+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 28914 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:17.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:17.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:17.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:17.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:17.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:17.684+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:17.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:17.686+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:17.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:17.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:17.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:17.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:17.935 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:17.740+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:17.936 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:17.742+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:18.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:17.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:18.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:17.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:18.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:17.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:18.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:17.936+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:19.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:18.751+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:19.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:18.751+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:19.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:18.751+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:19.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:18.752+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:19.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:18.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:19.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:18.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:19.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:18.715+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:19.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:18.716+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:19.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:18.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:19.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:18.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:19.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:18.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:19.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:18.896+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:19.665 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:18:20.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:19.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:20.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:19.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:20.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:19.862+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:20.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:19.863+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:20.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:19.731+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:20.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:19.731+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:20.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:19.731+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:20.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:19.732+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:20.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:19.716+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:20.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:19.716+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:20.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:19.716+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:20.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:19.718+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:21.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:20.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:21.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:20.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:21.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:20.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:21.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:20.832+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:21.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:20.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:21.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:20.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:21.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:20.764+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:21.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:20.766+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:21.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:20.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:21.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:20.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:21.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:20.730+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:21.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:20.731+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:22.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:21.760+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:22.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:21.760+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:22.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:21.760+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:22.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:21.762+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:22.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:21.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:22.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:21.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:22.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:21.750+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:22.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:21.751+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:22.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:21.797+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:22.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:21.797+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:22.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:21.797+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:22.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:21.798+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:22.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:18:22.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:18:22.737 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:18:22.738 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:18:22.738 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-19T19:18:22.738 DEBUG:teuthology.orchestra.run.smithi059:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-19T19:18:22.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:18:22.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:18:22.835 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:22.535+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29004 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:22.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:22.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:22.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:22.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:22.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:22.742+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:22.836 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:22.743+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:22.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:22.781+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:22.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:22.781+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:22.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:22.781+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:22.837 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:22.782+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:23.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:22.837+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:23.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:22.837+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:23.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:22.837+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:23.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:22.838+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:23.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:23.762+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:23.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:23.762+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:23.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:23.762+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:23.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:23.762+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:24.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:23.805+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:24.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:23.805+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:24.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:23.805+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:24.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:23.806+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:24.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:23.845+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:24.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:23.846+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:24.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:23.846+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:24.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:23.846+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:25.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:24.894+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:25.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:24.894+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:25.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:24.894+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:25.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:24.894+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:25.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:24.826+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:25.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:24.826+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:25.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:24.826+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:25.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:24.828+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:25.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:24.808+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:25.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:24.808+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:25.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:24.808+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:25.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:24.809+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:25.813 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:18:26.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:25.911+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:26.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:25.911+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:26.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:25.911+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:26.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:25.911+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:26.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:25.858+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:26.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:25.858+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:26.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:25.858+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:26.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:25.859+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:26.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:25.825+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:26.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:25.825+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:26.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:25.825+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:26.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:25.826+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:27.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:26.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:27.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:26.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:27.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:26.865+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:27.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:26.866+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:27.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:26.841+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:27.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:26.841+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:27.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:26.841+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:27.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:26.842+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:27.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:26.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:27.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:26.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:27.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:26.812+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:27.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:26.813+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:27.861 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:27.538+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29095 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:28.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:27.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:28.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:27.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:28.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:27.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:28.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:27.873+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:27.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:27.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:27.895+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:27.896+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:28.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:27.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:28.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:27.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:28.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:27.859+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:28.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:27.860+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:29.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:28.914+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:29.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:28.914+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:29.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:28.914+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:29.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:28.915+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:29.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:28.864+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:29.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:28.864+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:29.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:28.864+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:29.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:28.865+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:29.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:28.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:29.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:28.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:29.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:28.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:29.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:28.936+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:30.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:29.964+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:30.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:29.964+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:30.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:29.964+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:30.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:29.965+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:30.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:29.876+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:30.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:29.876+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:30.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:29.876+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:30.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:29.878+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:30.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:29.900+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:30.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:29.900+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:30.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:29.900+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:30.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:29.900+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:31.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:30.922+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:31.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:30.922+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:31.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:30.922+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:31.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:30.923+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:31.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:30.893+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:31.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:30.893+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:31.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:30.893+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:31.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:30.894+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:31.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:30.899+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:31.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:30.899+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:31.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:30.899+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:31.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:30.900+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:32.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:31.954+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:32.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:31.954+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:32.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:31.954+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:32.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:31.955+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:32.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:31.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:32.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:31.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:32.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:31.849+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:32.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:31.850+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:32.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:31.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:32.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:31.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:32.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:31.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:32.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:31.889+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:32.891 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:32.542+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29185 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:33.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:32.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:33.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:32.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:33.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:32.936+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:33.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:32.938+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:33.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:32.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:33.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:32.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:33.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:32.981+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:33.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:32.982+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:33.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:32.893+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:33.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:32.893+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:33.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:32.893+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:33.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:32.894+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:33.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:33.885+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:33.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:33.885+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:33.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:33.885+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:33.922 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:33.886+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:34.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:33.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:34.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:33.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:34.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:33.965+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:34.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:33.966+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:34.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:33.924+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:34.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:33.924+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:34.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:33.924+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:34.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:33.924+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:34.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:18:34.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:18:35.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:34.941+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:35.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:34.941+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:35.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:34.941+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:35.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:34.942+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:35.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:34.894+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:35.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:34.894+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:35.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:34.894+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:35.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:34.894+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:35.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:34.920+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:35.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:34.920+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:34.920+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:35.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:34.920+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:36.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:35.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:36.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:35.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:36.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:35.935+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:36.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:35.936+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:36.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:35.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:36.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:35.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:36.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:35.863+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:36.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:35.864+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:36.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:35.911+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:36.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:35.911+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:36.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:35.911+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:36.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:35.912+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:37.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:36.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:37.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:36.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:37.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:36.928+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:37.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:36.929+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:37.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:36.911+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:37.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:36.911+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:37.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:36.911+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:37.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:36.912+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:37.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:36.951+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:37.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:36.951+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:37.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:36.951+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:37.198 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:36.951+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:37.873 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:18:37.943 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:37.546+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29277 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:38.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:37.945+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:38.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:37.945+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:38.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:37.945+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:38.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:37.946+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:38.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:37.976+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:38.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:37.976+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:38.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:37.976+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:38.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:37.976+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:38.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:37.969+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:38.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:37.969+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:38.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:37.969+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:38.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:37.970+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:39.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:38.903+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:39.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:38.903+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:39.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:38.903+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:39.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:38.904+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:39.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:39.002+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:39.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:39.002+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:39.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:39.002+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:39.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:39.003+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:39.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:38.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:39.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:38.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:39.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:38.952+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:39.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:38.953+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:40.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:39.868+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:40.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:39.868+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:40.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:39.868+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:40.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:39.869+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:40.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:40.051+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:40.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:40.051+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:40.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:40.051+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:40.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:40.052+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:40.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:39.944+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:40.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:39.944+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:40.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:39.944+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:40.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:39.944+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:41.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:40.902+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:41.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:40.902+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:41.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:40.902+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:41.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:40.903+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:41.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:40.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:41.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:40.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:41.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:40.909+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:41.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:40.910+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:41.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:41.005+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:41.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:41.005+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:41.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:41.005+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:41.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:41.006+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:42.031 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:41.923+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:42.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:41.923+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:42.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:41.923+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:42.032 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:41.925+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:42.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:41.876+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:42.032 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:41.876+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:42.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:41.876+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:42.033 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:41.877+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:42.410 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:42.033+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:42.410 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:42.033+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:42.410 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:42.033+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:42.411 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:42.035+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:42.674 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:42.549+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29367 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:42.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:42.835+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:42.934 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:42.835+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:42.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:42.835+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:42.935 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:42.836+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:43.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:42.936+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:43.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:42.936+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:43.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:42.936+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:43.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:42.937+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:43.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:42.990+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:43.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:42.990+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:43.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:42.990+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:43.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:42.991+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:43.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:43.806+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:43.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:43.806+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:43.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:43.806+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:43.937 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:43.807+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:44.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:44.004+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:44.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:44.004+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:44.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:44.004+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:44.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:44.005+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:44.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:43.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:44.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:43.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:44.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:43.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:44.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:43.939+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:45.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:45.043+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:45.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:45.043+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:45.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:45.043+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:45.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:45.043+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:45.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:44.857+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:45.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:44.857+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:45.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:44.857+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:45.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:44.857+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:45.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:44.889+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:45.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:44.889+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:45.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:44.889+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:45.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:44.891+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:46.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:45.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:46.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:45.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:46.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:45.889+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:46.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:45.889+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:46.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:46.078+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:46.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:46.078+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:46.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:46.078+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:46.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:46.079+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:46.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:45.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:46.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:45.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:46.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:45.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:46.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:45.927+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:47.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:47.089+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:47.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:47.089+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:47.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:47.089+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:47.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:47.089+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:47.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:46.936+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:47.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:46.936+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:47.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:46.936+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:47.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:46.937+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:47.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:46.895+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:47.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:46.895+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:47.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:46.895+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:47.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:46.897+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:47.876 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:47.553+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29458 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:48.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:47.938+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:48.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:47.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:48.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:47.939+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:48.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:47.939+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:48.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:47.878+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:48.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:47.878+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:48.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:47.878+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:48.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:47.879+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:48.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:48.068+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:48.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:48.068+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:48.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:48.068+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:48.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:48.069+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:49.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:48.891+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:49.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:48.891+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:49.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:48.891+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:49.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:48.892+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:49.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:49.106+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:49.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:49.106+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:49.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:49.106+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:49.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:49.106+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:49.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:48.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:49.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:48.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:49.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:48.896+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:49.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:48.897+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:49.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:18:49.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:18:50.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:49.916+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:50.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:49.916+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:50.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:49.916+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:50.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:49.917+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:50.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:49.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:50.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:49.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:50.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:49.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:50.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:49.939+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:50.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:50.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:50.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:50.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:50.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:50.156+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:50.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:50.156+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:51.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:50.959+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:51.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:50.959+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:51.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:50.959+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:51.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:50.960+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:51.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:51.147+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:51.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:51.147+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:51.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:51.147+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:51.448 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:51.148+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:51.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:50.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:51.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:50.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:51.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:50.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:51.448 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:50.949+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:52.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:52.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:52.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:52.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:52.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:52.168+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:52.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:52.169+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:52.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:51.936+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:52.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:51.936+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:52.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:51.936+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:52.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:51.937+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:52.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:51.991+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:52.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:51.991+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:52.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:51.991+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:52.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:51.992+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:52.888 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:52.557+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29548 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:52.945 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:18:53.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:52.890+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:53.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:52.890+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:53.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:52.890+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:53.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:52.891+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:53.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:52.973+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:53.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:52.973+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:53.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:52.973+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:53.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:52.973+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:53.640 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:53.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:53.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:53.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:53.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:53.199+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:53.641 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:53.199+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:53.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:53.853+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:53.945 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:53.853+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:53.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:53.853+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:53.946 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:53.854+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:54.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:53.953+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:54.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:53.953+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:54.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:53.953+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:54.242 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:53.954+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:54.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:54.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:54.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:54.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:54.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:54.244+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:54.696 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:54.245+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:55.002 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:54.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:55.002 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:54.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:55.002 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:54.859+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:55.002 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:54.860+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:55.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:55.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:55.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:55.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:55.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:55.201+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:55.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:55.201+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:55.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:55.004+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:55.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:55.004+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:55.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:55.004+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:55.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:55.005+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:55.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:18:55.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:18:56.017 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:18:56.018 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:18:56.018 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-19T19:18:56.020 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270/remote/smithi059/crash 2023-12-19T19:18:56.021 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/crash -- . 2023-12-19T19:18:56.059 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270/remote/smithi089/crash 2023-12-19T19:18:56.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:18:56.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:18:56.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:55.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:56.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:55.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:56.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:55.872+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:56.180 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:55.872+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:56.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:55.989+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:56.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:55.989+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:56.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:55.989+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:56.181 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:55.990+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:56.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:56.182+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:56.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:56.182+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:56.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:56.182+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:56.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:56.183+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:57.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:56.824+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:57.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:56.824+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:57.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:56.824+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:57.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:56.826+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:57.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:56.941+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:57.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:56.941+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:57.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:56.941+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:57.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:56.942+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:57.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:57.160+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:57.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:57.160+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:57.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:57.160+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:57.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:57.161+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:57.846 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:18:57.561+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29640 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:18:58.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:57.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:58.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:57.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:58.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:57.949+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:58.186 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:57.950+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:58.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:57.848+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:58.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:57.848+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:58.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:57.848+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:58.186 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:57.849+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:58.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:58.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:58.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:58.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:58.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:58.186+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:58.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:58.186+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:59.125 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:18:59.175 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:58.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:59.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:58.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:59.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:58.886+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:18:59.176 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:58.888+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:18:59.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:58.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:59.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:58.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:59.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:58.984+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:18:59.177 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:58.985+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:18:59.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:59.178+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:59.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:59.178+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:59.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:59.178+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:18:59.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:18:59.178+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:00.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:59.976+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:00.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:59.976+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:00.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:59.976+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:00.153 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:18:59.977+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:00.153 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:59.839+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:00.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:59.839+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:00.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:59.839+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:00.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:18:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:18:59.840+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:00.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:00.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:00.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:00.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:00.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:00.155+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:00.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:00.155+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:01.114 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:00.975+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:01.114 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:00.975+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:01.114 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:00.975+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:01.114 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:00.976+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:01.115 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:00.880+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:01.115 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:00.880+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:01.115 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:00.880+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:01.115 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:00.881+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:01.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:01.116+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:01.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:01.116+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:01.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:01.116+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:01.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:01.117+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:01.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:01.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:01.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:01.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:01.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:01.924+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:01.993 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:01.924+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:02.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:02.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:02.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:02.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:02.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:02.096+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:02.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:02.097+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:02.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:01.995+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:02.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:01.995+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:02.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:01.995+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:02.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:01.996+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:02.945 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:02.564+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29730 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:03.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:02.949+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:03.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:02.949+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:03.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:02.950+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:03.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:02.951+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:03.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:03.112+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:03.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:03.112+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:03.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:03.112+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:03.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:03.112+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:03.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:03.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:03.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:03.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:03.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:03.000+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:03.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:03.001+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:04.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:04.015+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:04.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:04.015+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:04.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:04.015+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:04.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:04.016+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:04.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:03.987+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:04.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:03.987+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:04.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:03.987+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:04.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:03.988+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:04.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:04.079+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:04.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:04.079+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:04.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:04.079+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:04.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:04.079+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:05.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:04.990+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:05.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:04.990+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:05.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:04.990+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:05.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:04.991+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:05.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:05.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:05.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:05.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:05.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:05.016+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:05.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:05.016+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:05.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:05.104+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:05.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:05.104+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:05.448 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:05.104+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:05.448 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:05.105+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:06.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:06.013+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:06.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:06.013+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:06.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:06.013+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:06.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:06.014+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:06.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:06.024+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:06.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:06.024+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:06.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:06.024+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:06.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:06.025+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:06.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:06.137+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:06.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:06.137+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:06.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:06.137+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:06.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:06.138+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:07.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:07.002+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:07.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:07.002+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:07.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:07.002+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:07.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:07.003+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:07.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:07.088+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:07.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:07.088+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:07.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:07.088+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:07.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:07.089+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:07.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:07.005+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:07.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:07.005+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:07.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:07.005+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:07.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:07.007+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:07.945 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:07.568+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29819 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:08.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:19:08.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:19:08.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:08.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:08.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:08.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:08.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:08.012+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:08.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:08.013+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:08.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:08.050+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:08.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:08.050+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:08.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:08.050+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:08.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:08.050+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:08.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:08.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:08.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:08.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:08.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:08.052+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:08.448 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:08.052+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:09.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:08.997+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:09.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:08.997+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:09.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:08.997+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:09.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:08.998+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:09.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:09.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:09.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:09.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:09.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:09.032+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:09.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:09.032+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:09.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:09.038+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:09.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:09.038+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:09.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:09.038+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:09.447 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:09.039+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:10.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:10.044+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:10.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:10.044+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:10.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:10.044+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:10.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:10.044+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:10.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:10.048+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:10.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:10.048+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:10.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:10.048+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:10.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:10.048+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:10.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:10.011+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:10.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:10.011+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:10.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:10.011+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:10.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:10.012+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:11.189 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:19:11.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:11.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:11.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:11.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:11.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:11.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:11.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:11.078+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:11.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:10.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:11.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:10.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:11.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:10.985+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:11.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:10.985+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:11.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:10.999+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:11.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:10.999+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:11.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:10.999+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:11.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:10.999+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:12.098 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:11.966+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:12.099 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:11.966+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:12.099 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:11.966+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:12.099 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:11.967+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:12.099 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:11.976+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:12.099 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:11.976+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:12.100 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:11.976+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:12.100 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:11.977+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:12.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:12.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:12.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:12.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:12.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:12.101+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:12.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:12.102+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:12.938 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:12.571+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 29908 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:13.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:13.002+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:13.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:13.002+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:13.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:13.002+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:13.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:13.003+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:13.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:12.940+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:13.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:12.940+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:13.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:12.940+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:13.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:12.942+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:13.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:13.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:13.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:13.094+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:13.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:13.095+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:13.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:13.095+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:14.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:13.907+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:14.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:13.907+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:14.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:13.907+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:14.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:13.908+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:14.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:14.028+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:14.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:14.028+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:14.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:14.028+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:14.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:14.028+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:14.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:14.069+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:14.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:14.069+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:14.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:14.069+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:14.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:14.070+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:15.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:14.937+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:15.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:14.937+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:15.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:14.937+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:15.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:14.938+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:15.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:15.019+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:15.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:15.019+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:15.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:15.019+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:15.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:15.020+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:15.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:15.098+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:15.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:15.098+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:15.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:15.098+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:15.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:15.099+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:16.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:15.944+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:16.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:15.944+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:16.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:15.944+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:16.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:15.945+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:16.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:16.005+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:16.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:16.005+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:16.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:16.005+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:16.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:16.006+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:16.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:16.085+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:16.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:16.085+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:16.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:16.085+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:16.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:16.086+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:17.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:17.126+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:17.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:17.126+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:17.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:17.126+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:17.446 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:17.127+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:17.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:16.969+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:17.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:16.969+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:17.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:16.969+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:17.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:16.970+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:17.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:16.960+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:17.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:16.960+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:17.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:16.960+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:17.447 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:16.960+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:17.942 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:17.575+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30000 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:18.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:18.173+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:18.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:18.173+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:18.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:18.173+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:18.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:18.174+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:18.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:17.963+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:18.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:17.963+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:18.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:17.963+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:18.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:17.964+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:18.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:17.945+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:18.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:17.945+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:18.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:17.945+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:18.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:17.946+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:19.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:18.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:19.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:18.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:19.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:18.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:19.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:18.932+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:19.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:18.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:19.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:18.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:19.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:18.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:19.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:18.931+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:19.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:19.216+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:19.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:19.216+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:19.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:19.216+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:19.696 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:19.217+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:20.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:19.914+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:20.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:19.914+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:20.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:19.914+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:20.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:19.915+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:20.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:19.977+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:20.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:19.977+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:20.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:19.977+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:20.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:19.978+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:20.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:20.218+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:20.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:20.218+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:20.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:20.218+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:20.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:20.219+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:21.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:20.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:21.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:20.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:21.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:20.967+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:21.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:20.968+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:21.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:20.943+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:21.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:20.943+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:21.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:20.943+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:21.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:20.944+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:21.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:21.233+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:21.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:21.233+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:21.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:21.233+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:21.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:21.234+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:22.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:21.929+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:22.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:21.929+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:22.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:21.929+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:22.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:21.929+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:22.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:21.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:22.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:21.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:22.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:21.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:22.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:21.939+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:22.576 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:22.220+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:22.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:22.220+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:22.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:22.220+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:22.577 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:22.221+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:22.891 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:22.579+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30090 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:23.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:19:23.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:19:23.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:22.894+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:23.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:22.894+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:23.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:22.894+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:23.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:22.895+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:23.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:22.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:23.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:22.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:23.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:22.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:23.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:22.924+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:23.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:23.207+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:23.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:23.207+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:23.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:23.207+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:23.642 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:23.208+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:23.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:23.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:23.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:23.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:23.905 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:23.875+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:23.906 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:23.875+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:24.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:23.907+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:24.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:23.907+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:24.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:23.907+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:24.191 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:23.909+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:24.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:24.193+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:24.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:24.193+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:24.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:24.193+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:24.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:24.194+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:25.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:24.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:25.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:24.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:25.156 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:24.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:25.157 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:24.929+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:25.157 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:24.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:25.157 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:24.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:25.157 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:24.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:25.157 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:24.867+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:25.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:25.159+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:25.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:25.159+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:25.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:25.159+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:25.418 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:25.159+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:26.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:26.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:26.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:26.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:26.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:26.122+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:26.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:26.123+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:26.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:25.913+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:26.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:25.913+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:26.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:25.913+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:26.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:25.913+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:26.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:25.909+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:26.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:25.909+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:26.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:25.909+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:26.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:25.910+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:26.257 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:19:27.155 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:26.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:27.155 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:26.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:27.155 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:26.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:27.155 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:26.911+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:27.156 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:26.896+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:27.156 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:26.896+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:27.156 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:26.896+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:27.156 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:26.896+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:27.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:27.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:27.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:27.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:27.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:27.157+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:27.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:27.159+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:27.928 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:27.583+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30181 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:28.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:28.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:28.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:28.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:28.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:28.110+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:28.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:28.110+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:28.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:27.948+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:28.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:27.948+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:28.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:27.948+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:28.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:27.949+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:27.931+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:27.931+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:27.931+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:28.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:27.932+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:29.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:28.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:29.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:28.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:29.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:28.893+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:29.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:28.893+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:29.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:29.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:29.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:29.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:29.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:29.077+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:29.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:29.077+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:29.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:28.951+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:29.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:28.951+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:29.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:28.951+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:29.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:28.951+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:29.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:19:29.259 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:19:29.329 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:19:29.330 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:19:29.330 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-19T19:19:29.330 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T19:19:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:grep: /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph.log: No such file or directory 2023-12-19T19:19:29.370 INFO:tasks.cephadm:Compressing logs... 2023-12-19T19:19:29.370 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-19T19:19:29.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:19:29.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:19:29.442 INFO:teuthology.orchestra.run.smithi059.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-12-19T19:19:29.442 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-19T19:19:29.443 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi059.rtqzmc.log 2023-12-19T19:19:29.444 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.1.log 2023-12-19T19:19:29.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi059.rtqzmc.log: gzip -5 --verbose -- /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi059.abqklt.log 2023-12-19T19:19:29.444 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mgr.smithi059.ccbnob.log 2023-12-19T19:19:29.444 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.1.log: /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi059.abqklt.log: gzip -5 --verbose -- /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.0.log 2023-12-19T19:19:29.445 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mgr.smithi059.ccbnob.log: gzip -5 --verbose -- /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.2.log 2023-12-19T19:19:29.445 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-19T19:19:29.445 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mon.smithi059.log 2023-12-19T19:19:29.452 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-volume.log: /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.2.log: 90.8% -- replaced with /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-19T19:19:29.454 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mon.smithi059.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-19T19:19:29.479 INFO:teuthology.orchestra.run.smithi059.stderr: 95.0% -- replaced with /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi059.abqklt.log.gz 2023-12-19T19:19:30.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:29.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:30.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:29.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:30.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:29.910+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:30.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:29.911+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:30.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:29.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:30.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:29.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:30.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:29.930+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:08:59.771279+0000 (oldest deadline 2023-12-19T19:09:19.771279+0000) 2023-12-19T19:19:30.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:29.931+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:30.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:30.113+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:30.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:30.113+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:30.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:30.113+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:30.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:30.114+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mgr.smithi059.ccbnob.log: file size changed while zipping 2023-12-19T19:19:30.218 INFO:teuthology.orchestra.run.smithi059.stderr: 89.9% -- replaced with /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mgr.smithi059.ccbnob.log.gz 2023-12-19T19:19:30.841 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mon.smithi059.log: file size changed while zipping 2023-12-19T19:19:30.841 INFO:teuthology.orchestra.run.smithi059.stderr: 92.5% -- replaced with /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mon.smithi059.log.gz 2023-12-19T19:19:31.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:30.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:31.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:30.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:31.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:30.928+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:31.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:30.929+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:31.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:30.884+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:31.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:31.127+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:31.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:31.127+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:31.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:31.127+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:31.197 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:31.128+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:32.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:31.835+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:32.143 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:31.978+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:32.143 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:31.978+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:32.143 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:31.978+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:32.143 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:31.979+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:32.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:32.144+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:32.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:32.144+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:32.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:32.144+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:32.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:32.145+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:32.464 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.2.log: file size changed while zipping 2023-12-19T19:19:32.464 INFO:teuthology.orchestra.run.smithi059.stderr: 91.8% -- replaced with /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.2.log.gz 2023-12-19T19:19:32.469 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:19:32.874 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:32.585+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30271 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:33.141 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:32.961+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:33.141 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:32.961+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:33.142 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:32.961+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:33.142 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:32.962+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:33.142 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:32.875+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:33.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:33.144+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:33.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:33.144+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:33.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:33.144+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:33.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:33.145+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:34.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:33.907+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:34.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:34.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:34.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:34.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:34.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:34.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:34.198 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:34.117+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:34.198 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:33.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:34.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:33.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:34.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:33.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:34.199 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:33.932+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:34.560 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.0.log: file size changed while zipping 2023-12-19T19:19:34.560 INFO:teuthology.orchestra.run.smithi059.stderr: 94.9% -- replaced with /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.0.log.gz 2023-12-19T19:19:35.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:34.902+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:35.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:35.105+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:35.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:35.105+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:35.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:35.105+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:35.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:35.106+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:35.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:34.936+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:35.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:34.936+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:35.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:34.936+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:35.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:34.938+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:36.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:35.873+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:36.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:36.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:36.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:36.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:36.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:36.067+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:36.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:36.069+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:36.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:35.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:36.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:35.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:36.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:35.926+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:36.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:35 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:35.927+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:37.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:36.886+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:37.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:37.085+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:37.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:37.085+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:37.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:37.085+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:37.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:37.086+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:37.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:36.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:37.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:36.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:37.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:36.938+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:37.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:36 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:36.939+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:37.238 INFO:teuthology.orchestra.run.smithi059.stderr:gzip: /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.1.log: file size changed while zipping 2023-12-19T19:19:37.238 INFO:teuthology.orchestra.run.smithi059.stderr: 96.3% -- replaced with /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-osd.1.log.gz 2023-12-19T19:19:37.854 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:37.590+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30363 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:38.114 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:37.857+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:38.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:37.941+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:38.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:37.941+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:38.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:37.941+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:38.115 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:37 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:37.942+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:38.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:38.117+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:38.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:38.117+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:38.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:38.117+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:38.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:38.118+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:39.149 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:38.872+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:39.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:38.943+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:39.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:38.943+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:39.149 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:38.943+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.972665+0000 (oldest deadline 2023-12-19T19:09:28.972665+0000) 2023-12-19T19:19:39.150 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:38 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:38.944+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:39.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:39.151+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:39.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:39.151+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:39.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:39.151+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:39.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:39.152+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:40.197 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:39.937+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:40.197 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:39 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:39.830+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:40.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:40.202+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:40.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:40.202+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:40.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:40.202+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:40.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:40.203+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:41.163 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:40.921+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:41.163 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:40 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:40.867+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:41.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:41.166+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:41.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:41.166+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:41.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:41.166+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:41.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:41.167+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:41.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:19:41.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:19:42.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:41.960+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:42.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:41 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:41.840+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:42.590 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:42.203+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:42.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:42.203+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:42.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:42.203+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:42.591 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:42.203+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:42.845 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:42.593+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30451 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:43.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:42.848+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:43.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:42 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:42.930+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:43.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:43.251+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:43.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:43.251+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:43.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:43.251+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:09:08.910398+0000 (oldest deadline 2023-12-19T19:09:28.910398+0000) 2023-12-19T19:19:43.643 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:43.252+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:43.945 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:43.891+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:43.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:43 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:43.827+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:44.533 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:19:44.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:44.256+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:45.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:44.930+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:45.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:44 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:44.819+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:45.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:45.249+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:46.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:45.903+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:46.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:45 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:45.772+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:46.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:46.270+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:47.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:46.935+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:47.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:46 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:46.814+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:47.593 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:47.245+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:47.944 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:47.596+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30542 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:47.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:47.840+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:48.272 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:47 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:47.979+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:48.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:48.275+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:49.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:49.024+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:49.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:48 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:48.867+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:49.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:49.255+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:50.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:49.983+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:50.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:49 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:49.824+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:50.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:50.243+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:51.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:50.952+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:51.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:50 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:50.803+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:51.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:51.199+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:52.183 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:51.827+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:52.183 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:51 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:51.921+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:52.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:52.186+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:52.536 INFO:teuthology.orchestra.run.smithi059.stderr: 91.9% -- replaced with /var/log/ceph/bf854274-9e9d-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi059.rtqzmc.log.gz 2023-12-19T19:19:52.537 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-19T19:19:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m23.113s 2023-12-19T19:19:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m40.156s 2023-12-19T19:19:52.538 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m0.955s 2023-12-19T19:19:52.816 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:52.600+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30632 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:53.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:52.911+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:53.174 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:52 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:52.819+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:53.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:53.177+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:53.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:53.787+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:53.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:53.787+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:53.916 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:53.787+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:53.917 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:53.788+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:54.173 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:53 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:53.919+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:54.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:54.176+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:55.160 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:54.930+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:55.160 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:54.800+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:55.160 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:54.800+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:55.160 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:54.800+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:55.161 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:54 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:54.800+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:55.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:55.163+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:56.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:55.954+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:56.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:55.826+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:56.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:55.826+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:56.138 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:55.826+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:56.138 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:55 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:55.826+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:56.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:56.153+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:56.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:19:56.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:19:57.176 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:56.912+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:57.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:56.808+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:57.176 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:56.808+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:57.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:56.808+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:57.177 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:56 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:56.808+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:57.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:57.179+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:57.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:19:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:19:57.604+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30724 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:19:57.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:57.844+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:57.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:57.844+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:57.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:57.844+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:57.890 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:57.844+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:58.174 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:57 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:57.892+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:58.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:58.177+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:59.167 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:58.854+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:19:59.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:58.852+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:59.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:58.852+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:59.167 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:58.852+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:19:59.168 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:58 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:58.853+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:59.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:19:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:19:59.170+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:19:59.601 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:20:00.154 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:19:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:19:59.872+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:00.154 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:00.157+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:00.155 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:59.810+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:00.155 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:59.810+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:00.155 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:59.810+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:00.155 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:19:59 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:19:59.810+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:01.111 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:00.857+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:01.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:00.852+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:01.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:00.852+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:01.111 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:00.852+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:01.112 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:00 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:00.853+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:01.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:01.114+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:02.137 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:01.879+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:02.137 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:01.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:02.138 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:01.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:02.138 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:01.867+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:02.138 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:01 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:01.867+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:02.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:02.140+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:02.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:20:02.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:20:02.673 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:20:02.674 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:20:02.674 INFO:tasks.cephadm:Removing cluster... 2023-12-19T19:20:02.674 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf854274-9e9d-11ee-95a5-87774f69a715 --force 2023-12-19T19:20:02.889 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:20:02.607+0000 7f9274058700 -1 mon.smithi059@0(probing) e2 get_health_metrics reporting 30814 slow ops, oldest is log(1 entries from seq 292 at 2023-12-19T18:47:48.565938+0000) 2023-12-19T19:20:02.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:02.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:02.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:02.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:02.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:02.832+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:02.889 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:02.833+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:03.144 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:03 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for bf854274-9e9d-11ee-95a5-87774f69a715... 2023-12-19T19:20:03.145 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:02 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:02.892+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:03.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:03.146+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:03.446 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:20:03.304+0000 7f927705e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T19:20:03.446 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059[80657]: 2023-12-19T19:20:03.304+0000 7f927705e700 -1 mon.smithi059@0(probing) e2 *** Got Signal Terminated *** 2023-12-19T19:20:03.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:03 smithi059 podman[135665]: 2023-12-19 19:20:03.539368101 +0000 UTC m=+0.343768610 container died 297c7051711f656f92e79be4daf1e761c5efbcb116487e3e7a0307c567c60f4d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, RELEASE=quincy-a88f3ed, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2023-12-19T19:20:04.113 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:03.874+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:04.114 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:03.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:04.114 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:03.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:04.114 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:03.796+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:04.114 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:03 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:03.796+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:04.115 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:03 smithi059 podman[135665]: 2023-12-19 19:20:03.922086245 +0000 UTC m=+0.726486740 container cleanup 297c7051711f656f92e79be4daf1e761c5efbcb116487e3e7a0307c567c60f4d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2023-12-19T19:20:04.115 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:03 smithi059 bash[135665]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059 2023-12-19T19:20:04.444 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:04 smithi059 podman[135679]: 2023-12-19 19:20:04.273872557 +0000 UTC m=+0.731575083 container remove 297c7051711f656f92e79be4daf1e761c5efbcb116487e3e7a0307c567c60f4d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-mon-smithi059, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, RELEASE=quincy-a88f3ed, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20231212, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2023-12-19T19:20:04.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:04.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:04.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:04.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:04.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:04.116+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:04.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:04.117+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:04.797 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:04 smithi059 systemd[1]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mon.smithi059.service: Deactivated successfully. 2023-12-19T19:20:04.797 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:04 smithi059 systemd[1]: Stopped Ceph mon.smithi059 for bf854274-9e9d-11ee-95a5-87774f69a715. 2023-12-19T19:20:04.797 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 19 19:20:04 smithi059 systemd[1]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715@mon.smithi059.service: Consumed 23.783s CPU time. 2023-12-19T19:20:05.047 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:04.852+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:05.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:04.852+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:05.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:04.852+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:05.048 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:04.853+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:05.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:04.799+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:05.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:04.799+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:05.048 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:04.799+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:05.049 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:04 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:04.800+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:05.333 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:05.162+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:05.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:05.162+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:05.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:05.162+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:05.334 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:05.163+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:06.116 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:05.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:06.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:05.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:06.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:05.854+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:06.117 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:05.854+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:06.117 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:05.843+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:06.117 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:05.843+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:06.117 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:05.843+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:06.118 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:05 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:05.843+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:06.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:06.182+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:06.438 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:06.182+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:06.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:06.182+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:06.439 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:06.183+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:07.131 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:06.889+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:07.131 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:06.889+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:07.132 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:06.889+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:07.132 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:06.890+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:07.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:06.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:07.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:06.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:07.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:06.889+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:07.132 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:06 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:06.890+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:07.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:07.200+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:07.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:07.200+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:07.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:07.200+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:07.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:07.201+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:08.061 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:07.902+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:08.061 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:07.902+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:08.061 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:07.902+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:08.061 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:07.903+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:08.061 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:07.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:08.062 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:07.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:08.062 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:07.923+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:08.062 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:07 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:07.923+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:08.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:08.172+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:08.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:08.172+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:08.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:08.172+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:08.403 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:08.173+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:09.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:08.916+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:09.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:08.916+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:09.015 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:08.916+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:09.016 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:08.917+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:09.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:08.943+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:09.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:08.943+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:09.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:08.943+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:09.016 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:08 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:08.944+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:09.308 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:09.179+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:09.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:09.179+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:09.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:09.179+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:09.309 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:09.180+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:10.077 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:09.942+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:10.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:09.942+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:10.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:09.942+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:10.078 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:09.942+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:10.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:09.955+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:10.078 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:09.955+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:10.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:09.955+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:10.079 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:09 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:09.956+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:10.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:10.137+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:10.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:10.137+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:10.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:10.137+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:10.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:10.138+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:11.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:11.132+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:11.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:11.132+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:11.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:11.132+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:11.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:11.133+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:11.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:10.906+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:11.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:10.906+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:11.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:10.906+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:11.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:10.907+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:11.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:10.990+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:11.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:10.991+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:11.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:10.991+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:11.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:10 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:10.991+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:12.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:11.969+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:12.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:11.969+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:12.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:11.969+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:12.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:11.969+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:12.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:12.112+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:12.195 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:12.112+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:12.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:12.112+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:12.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:12.113+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:12.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:11.944+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:12.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:11.944+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:12.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:11.944+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:12.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:11 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:11.945+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:13.158 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:12 smithi059 systemd[1]: Stopping Ceph osd.0 for bf854274-9e9d-11ee-95a5-87774f69a715... 2023-12-19T19:20:13.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:12.898+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:13.158 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:12.898+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:13.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:12.898+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:13.159 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:12.899+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:13.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:12.962+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:13.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:12.962+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:13.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:12.962+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:13.159 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:12 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:12.963+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:13.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:13.161+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:13.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:13.161+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:13.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:13.161+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:13.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:13.161+0000 7f4b20364700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T19:20:13.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:13.161+0000 7f4b20364700 -1 osd.0 47 *** Got signal Terminated *** 2023-12-19T19:20:13.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:13.161+0000 7f4b20364700 -1 osd.0 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-19T19:20:13.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:13.162+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:14.143 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:13.885+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:14.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:13.885+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:14.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:13.885+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:14.144 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:13.886+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:14.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:13.992+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:14.144 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:13.992+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:14.145 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:13.992+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:14.145 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:13 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:13.992+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:14.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:14.147+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:14.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:14.147+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:14.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:14.147+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:14.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:14.148+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:15.184 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:15.030+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:15.184 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:15.030+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:15.184 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:15.030+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:15.184 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:15.031+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:15.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:14.883+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:15.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:14.883+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:15.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:14.883+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:15.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:14 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:14.884+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:15.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:15.187+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:15.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:15.187+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:15.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:15.187+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:15.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:15.188+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:16.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:15.918+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:16.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:15.918+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:16.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:15.918+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:16.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:15 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:15.919+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:16.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:16.075+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:16.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:16.075+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:16.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:16.075+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:16.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:16.076+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:16.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:16.198+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:16.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:16.198+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:16.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:16.198+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:16.695 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:16.199+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:17.444 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:17.188+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:17.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:17.188+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:17.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:17.188+0000 7f4b1995c700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:43.569551+0000 (oldest deadline 2023-12-19T19:20:03.569551+0000) 2023-12-19T19:20:17.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0[99458]: 2023-12-19T19:20:17.190+0000 7f4b1995c700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14524.0:6859 3.6 3:638532d9:::10000004ea3.00000000:head [write 0~1043 [1@-1] in=1043b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:17.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:16.965+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:17.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:16.965+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:17.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:16.965+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:17.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:16 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:16.966+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:17.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:17.106+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:17.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:17.106+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:17.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:17.106+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:17.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:17.107+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:18.194 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:17.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:18.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:17.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:18.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:17.931+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:18.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:17 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:17.932+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:18.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:18.108+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:18.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:18.108+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:18.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:18.108+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:18.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:18.108+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:18.908 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:18 smithi059 podman[136501]: 2023-12-19 19:20:18.625158228 +0000 UTC m=+5.583765746 container died 87e75bce1c5bab963c48121b91fc577325f497712fe735a4f2fcb005e4f8b15c (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-a88f3ed, org.label-schema.build-date=20231212, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_CLEAN=True, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-19T19:20:19.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:19 smithi059 podman[136501]: 2023-12-19 19:20:19.012104022 +0000 UTC m=+5.970711541 container cleanup 87e75bce1c5bab963c48121b91fc577325f497712fe735a4f2fcb005e4f8b15c (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-a88f3ed, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2023-12-19T19:20:19.194 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:19 smithi059 bash[136501]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0 2023-12-19T19:20:19.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:18.977+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:19.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:18.977+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:19.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:18.977+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:19.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:18 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:18.978+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:19.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:19.149+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:19.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:19.149+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:19.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:19.149+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:19.196 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:19 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:19.150+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:19.644 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:19 smithi059 podman[136523]: 2023-12-19 19:20:19.280434224 +0000 UTC m=+0.650904887 container remove 87e75bce1c5bab963c48121b91fc577325f497712fe735a4f2fcb005e4f8b15c (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2023-12-19T19:20:19.896 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:19 smithi059 podman[136611]: 2023-12-19 19:20:19.551535566 +0000 UTC m=+0.021530664 image pull f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd 2023-12-19T19:20:19.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:19 smithi059 podman[136611]: 2023-12-19 19:20:19.681383154 +0000 UTC m=+0.151378281 container create 2ce52e56b692d7cd66203b72b31679a3db96fa1f8a962e39a7f83aad03f2db97 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0-deactivate, RELEASE=quincy-a88f3ed, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2023-12-19T19:20:19.897 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:19 smithi059 podman[136611]: 2023-12-19 19:20:19.901140537 +0000 UTC m=+0.371135619 container init 2ce52e56b692d7cd66203b72b31679a3db96fa1f8a962e39a7f83aad03f2db97 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-a88f3ed, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2023-12-19T19:20:20.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:20.147+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:20.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:20.147+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:20.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:20.147+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:20.195 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:20.148+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:20.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:20.001+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:20.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:20.001+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:20.195 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:20.001+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:20.196 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:20 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:20.002+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:20.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:19 smithi059 podman[136611]: 2023-12-19 19:20:19.904512119 +0000 UTC m=+0.374507203 container start 2ce52e56b692d7cd66203b72b31679a3db96fa1f8a962e39a7f83aad03f2db97 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, maintainer=Guillaume Abrioux , RELEASE=quincy-a88f3ed, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2) 2023-12-19T19:20:20.196 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:20 smithi059 podman[136611]: 2023-12-19 19:20:20.009100267 +0000 UTC m=+0.479095354 container attach 2ce52e56b692d7cd66203b72b31679a3db96fa1f8a962e39a7f83aad03f2db97 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-a88f3ed, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.31.3, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2023-12-19T19:20:20.694 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:20 smithi059 podman[136611]: 2023-12-19 19:20:20.290005413 +0000 UTC m=+0.760000502 container died 2ce52e56b692d7cd66203b72b31679a3db96fa1f8a962e39a7f83aad03f2db97 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0-deactivate, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-a88f3ed, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2023-12-19T19:20:21.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.020+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:21.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.020+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:21.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.020+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:21.102 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.021+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:21.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 systemd[1]: Stopping Ceph osd.1 for bf854274-9e9d-11ee-95a5-87774f69a715... 2023-12-19T19:20:21.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:21 smithi059 podman[136639]: 2023-12-19 19:20:21.140400777 +0000 UTC m=+0.840354255 container remove 2ce52e56b692d7cd66203b72b31679a3db96fa1f8a962e39a7f83aad03f2db97 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-a88f3ed, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20231212, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2023-12-19T19:20:21.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:21 smithi059 systemd[1]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.0.service: Deactivated successfully. 2023-12-19T19:20:21.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:21 smithi059 systemd[1]: Stopped Ceph osd.0 for bf854274-9e9d-11ee-95a5-87774f69a715. 2023-12-19T19:20:21.445 INFO:journalctl@ceph.osd.0.smithi059.stdout:Dec 19 19:20:21 smithi059 systemd[1]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.0.service: Consumed 15min 49.512s CPU time. 2023-12-19T19:20:21.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:21.105+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:21.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:21.105+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:21.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:21.105+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:21.446 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:21.106+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:21.944 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.682+0000 7f925c98c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T19:20:21.944 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.682+0000 7f925c98c700 -1 osd.1 47 *** Got signal Terminated *** 2023-12-19T19:20:21.944 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.682+0000 7f925c98c700 -1 osd.1 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-19T19:20:22.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:22.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:22.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:22.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:22.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:22.124+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:22.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:22 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:22.125+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:22.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.990+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:22.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.990+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:22.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.990+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:22.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:21 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:21.991+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:23.345 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:23.036+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:23.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:23.037+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:23.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:23.037+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:23.346 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:23.037+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:23.347 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:23.123+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:23.347 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:23.123+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:23.347 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:23.123+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:23.347 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:23 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:23.124+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:24.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:24.073+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:24.354 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:24.073+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:24.355 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:24.073+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:24.355 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:24.074+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:24.355 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:24.113+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:24.355 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:24.113+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:24.355 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:24.113+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:24.355 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:24 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:24.114+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:25.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:25.161+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:25.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:25.161+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:25.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:25.161+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:25.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:25.161+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:25.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:25.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:25.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:25.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:25.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:25.087+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:25.445 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:25 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:25.088+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:26.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:26.196+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:26.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:26.196+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:26.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:26.196+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:26.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:26.196+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:26.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:26.063+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:26.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:26.063+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:26.446 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:26.063+0000 7f925a78d700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:44.541203+0000 (oldest deadline 2023-12-19T19:20:04.541203+0000) 2023-12-19T19:20:26.447 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:26 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1[106908]: 2023-12-19T19:20:26.064+0000 7f925a78d700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:710 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=150692b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e47) 2023-12-19T19:20:27.430 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:27 smithi059 podman[136726]: 2023-12-19 19:20:27.161954735 +0000 UTC m=+5.660093447 container died 1b133ef58db0f0ef49c2781a7e2581dabe877db08f5bc08b42c63072b9e76fbd (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-a88f3ed, org.label-schema.schema-version=1.0, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3) 2023-12-19T19:20:27.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:27.231+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:27.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:27.231+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:27.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:27.231+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:27.431 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:27 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:27.231+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:27.694 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:27 smithi059 podman[136726]: 2023-12-19 19:20:27.499805282 +0000 UTC m=+5.997943980 container cleanup 1b133ef58db0f0ef49c2781a7e2581dabe877db08f5bc08b42c63072b9e76fbd (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, GIT_CLEAN=True, org.label-schema.build-date=20231212, RELEASE=quincy-a88f3ed, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-19T19:20:27.695 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:27 smithi059 bash[136726]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1 2023-12-19T19:20:28.185 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:27 smithi059 podman[136745]: 2023-12-19 19:20:27.78438767 +0000 UTC m=+0.624260650 container remove 1b133ef58db0f0ef49c2781a7e2581dabe877db08f5bc08b42c63072b9e76fbd (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-a88f3ed, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3) 2023-12-19T19:20:28.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:28 smithi059 podman[136839]: 2023-12-19 19:20:28.158306592 +0000 UTC m=+0.018041738 image pull f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd 2023-12-19T19:20:28.444 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:28 smithi059 podman[136839]: 2023-12-19 19:20:28.302109417 +0000 UTC m=+0.161844548 container create 48a10f2a84cfc151c57c3c2cd0bdf6f00f3ac6565472fba82d23b2173914758f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-a88f3ed, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=) 2023-12-19T19:20:28.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:28.189+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:28.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:28.189+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:28.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:28.189+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:28.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:28 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:28.189+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:28.803 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:28 smithi059 podman[136839]: 2023-12-19 19:20:28.513111102 +0000 UTC m=+0.372846240 container init 48a10f2a84cfc151c57c3c2cd0bdf6f00f3ac6565472fba82d23b2173914758f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-a88f3ed, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-19T19:20:28.804 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:28 smithi059 podman[136839]: 2023-12-19 19:20:28.51681247 +0000 UTC m=+0.376547593 container start 48a10f2a84cfc151c57c3c2cd0bdf6f00f3ac6565472fba82d23b2173914758f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=quincy-a88f3ed, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2023-12-19T19:20:28.804 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:28 smithi059 podman[136839]: 2023-12-19 19:20:28.587983768 +0000 UTC m=+0.447718912 container attach 48a10f2a84cfc151c57c3c2cd0bdf6f00f3ac6565472fba82d23b2173914758f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-a88f3ed, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, GIT_BRANCH=HEAD) 2023-12-19T19:20:29.129 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:28 smithi059 podman[136839]: 2023-12-19 19:20:28.80814795 +0000 UTC m=+0.667883090 container died 48a10f2a84cfc151c57c3c2cd0bdf6f00f3ac6565472fba82d23b2173914758f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=quincy-a88f3ed, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-19T19:20:29.444 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:29.215+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:29.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:29.215+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:29.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:29.215+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:29.445 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:29.215+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:29.924 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:29 smithi059 systemd[1]: Stopping Ceph osd.2 for bf854274-9e9d-11ee-95a5-87774f69a715... 2023-12-19T19:20:29.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:29 smithi059 podman[136865]: 2023-12-19 19:20:29.499639196 +0000 UTC m=+0.686341385 container remove 48a10f2a84cfc151c57c3c2cd0bdf6f00f3ac6565472fba82d23b2173914758f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-1-deactivate, RELEASE=quincy-a88f3ed, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2023-12-19T19:20:29.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:29 smithi059 systemd[1]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.1.service: Deactivated successfully. 2023-12-19T19:20:29.924 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:29 smithi059 systemd[1]: Stopped Ceph osd.1 for bf854274-9e9d-11ee-95a5-87774f69a715. 2023-12-19T19:20:29.925 INFO:journalctl@ceph.osd.1.smithi059.stdout:Dec 19 19:20:29 smithi059 systemd[1]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.1.service: Consumed 31min 7.196s CPU time. 2023-12-19T19:20:30.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:29.927+0000 7f5070243700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T19:20:30.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:29.927+0000 7f5070243700 -1 osd.2 47 *** Got signal Terminated *** 2023-12-19T19:20:30.194 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:29 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:29.927+0000 7f5070243700 -1 osd.2 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-19T19:20:30.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:30.225+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:30.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:30.225+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:30.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:30.225+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:30.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:30 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:30.225+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:31.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:31.237+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:31.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:31.237+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:31.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:31.237+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:31.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:31 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:31.237+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:32.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:32.246+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:32.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:32.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:32.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:32.247+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:32.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:32 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:32.247+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:33.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:33.260+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:33.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:33.260+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:33.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:33.260+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:33.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:33 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:33.261+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:34.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:34.295+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:34.694 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:34.295+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:34.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:34.295+0000 7f506d843700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.89:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:19:33.739824+0000 (oldest deadline 2023-12-19T19:19:53.739824+0000) 2023-12-19T19:20:34.695 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:34 smithi059 ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2[114236]: 2023-12-19T19:20:34.295+0000 7f506d843700 -1 osd.2 47 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.14524.0:6861 3.1e 3:7fa22549:::10000004ea5.00000000:head [write 0~1071 [1@-1] in=1071b] snapc 1=[] ondisk+write+known_if_redirected e47) 2023-12-19T19:20:35.691 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:35 smithi059 podman[136943]: 2023-12-19 19:20:35.34601602 +0000 UTC m=+5.511809791 container died 0f56936ea44c8f80bd4668c9ad711fba1a45042a62602548101ce7245150992f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, maintainer=Guillaume Abrioux , RELEASE=quincy-a88f3ed, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0) 2023-12-19T19:20:35.944 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:35 smithi059 podman[136943]: 2023-12-19 19:20:35.777588255 +0000 UTC m=+5.943382010 container cleanup 0f56936ea44c8f80bd4668c9ad711fba1a45042a62602548101ce7245150992f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2, RELEASE=quincy-a88f3ed, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2023-12-19T19:20:35.944 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:35 smithi059 bash[136943]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2 2023-12-19T19:20:36.384 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:35 smithi059 podman[136960]: 2023-12-19 19:20:35.962047509 +0000 UTC m=+0.614415553 container remove 0f56936ea44c8f80bd4668c9ad711fba1a45042a62602548101ce7245150992f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2023-12-19T19:20:36.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:36 smithi059 podman[137050]: 2023-12-19 19:20:36.292060953 +0000 UTC m=+0.018076587 image pull f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd 2023-12-19T19:20:36.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:36 smithi059 podman[137050]: 2023-12-19 19:20:36.45450025 +0000 UTC m=+0.180515864 container create 826cfff95862b831ac7f7077b573bf3f5896681fedbe531489048845595455a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-a88f3ed, io.buildah.version=1.31.3, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD) 2023-12-19T19:20:36.678 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:36 smithi059 podman[137050]: 2023-12-19 19:20:36.682711917 +0000 UTC m=+0.408727547 container init 826cfff95862b831ac7f7077b573bf3f5896681fedbe531489048845595455a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, ceph=True, maintainer=Guillaume Abrioux ) 2023-12-19T19:20:36.944 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:36 smithi059 podman[137050]: 2023-12-19 19:20:36.686168038 +0000 UTC m=+0.412183652 container start 826cfff95862b831ac7f7077b573bf3f5896681fedbe531489048845595455a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-a88f3ed, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2023-12-19T19:20:36.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:36 smithi059 podman[137050]: 2023-12-19 19:20:36.765107627 +0000 UTC m=+0.491123266 container attach 826cfff95862b831ac7f7077b573bf3f5896681fedbe531489048845595455a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, GIT_CLEAN=True, RELEASE=quincy-a88f3ed, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True) 2023-12-19T19:20:37.263 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:36 smithi059 podman[137050]: 2023-12-19 19:20:36.982466832 +0000 UTC m=+0.708482457 container died 826cfff95862b831ac7f7077b573bf3f5896681fedbe531489048845595455a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , RELEASE=quincy-a88f3ed, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-19T19:20:37.944 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:37 smithi059 podman[137077]: 2023-12-19 19:20:37.576442047 +0000 UTC m=+0.588779481 container remove 826cfff95862b831ac7f7077b573bf3f5896681fedbe531489048845595455a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-bf854274-9e9d-11ee-95a5-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-a88f3ed, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2023-12-19T19:20:37.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:37 smithi059 systemd[1]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.2.service: Deactivated successfully. 2023-12-19T19:20:37.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:37 smithi059 systemd[1]: Stopped Ceph osd.2 for bf854274-9e9d-11ee-95a5-87774f69a715. 2023-12-19T19:20:37.945 INFO:journalctl@ceph.osd.2.smithi059.stdout:Dec 19 19:20:37 smithi059 systemd[1]: ceph-bf854274-9e9d-11ee-95a5-87774f69a715@osd.2.service: Consumed 20.849s CPU time. 2023-12-19T19:20:43.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:20:43.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:20:46.101 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:20:55.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:20:55.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:20:58.161 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:21:10.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:21:10.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:21:13.237 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:21:16.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:21:16.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:21:16.309 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:21:16.310 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:21:16.310 INFO:tasks.cephadm:Teardown complete 2023-12-19T19:21:16.310 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1540, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1794, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1797, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1673, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 178, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 297, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi089 2023-12-19T19:21:16.343 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-19T19:21:16.369 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-19T19:21:16.389 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-19T19:21:16.389 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-19T19:21:16.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:21:16.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:21:19.473 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:21:28.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:21:28.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:21:31.538 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:21:43.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:21:43.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:21:46.609 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:21:49.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:21:49.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:21:49.681 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:21:49.682 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:21:49.738 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi089 2023-12-19T19:21:49.760 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-19T19:21:49.775 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-19T19:21:49.776 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T19:21:49.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:21:49.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:21:49.800 INFO:teuthology.orchestra.run.smithi059.stderr:bash: line 1: ntpq: command not found 2023-12-19T19:21:49.805 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-19T19:21:49.805 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2023-12-19T19:21:49.805 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 118 +224us[ +185us] +/- 73ms 2023-12-19T19:21:49.805 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -268us[ -277us] +/- 72ms 2023-12-19T19:21:49.805 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 119 +116us[ +76us] +/- 90ms 2023-12-19T19:21:49.806 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:21:52.853 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:22:01.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:22:01.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:22:04.917 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:22:16.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:22:16.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:22:19.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:22:22.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:22:22.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:22:23.057 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:22:23.058 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:22:23.058 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi089 2023-12-19T19:22:23.079 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-19T19:22:23.093 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-19T19:22:23.093 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-19T19:22:23.107 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-19T19:22:23.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:22:23.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:22:26.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:22:35.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:22:35.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:22:38.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:22:50.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:22:50.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:22:53.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:22:56.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:22:56.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:22:56.501 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:22:56.502 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:22:56.502 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi089 2023-12-19T19:22:56.510 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-19T19:22:56.522 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-19T19:22:56.548 INFO:teuthology.task.internal:Duration was 3183.006181 seconds 2023-12-19T19:22:56.548 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-19T19:22:56.558 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-19T19:22:56.559 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-19T19:22:56.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:22:56.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:22:56.620 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-19T19:22:59.633 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:23:08.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:23:08.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:23:11.697 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:23:23.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:23:23.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:23:26.769 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:23:29.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:23:29.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:23:29.841 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:23:29.842 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:23:29.842 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi089 2023-12-19T19:23:29.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-19T19:23:29.870 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-19T19:23:29.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-19T19:23:29.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:23:29.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:23:32.977 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:23:41.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:23:41.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:23:45.041 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:23:57.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:23:57.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:24:00.117 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:24:03.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:24:03.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:24:03.185 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:24:03.186 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:24:03.186 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi089 2023-12-19T19:24:03.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-19T19:24:03.213 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-19T19:24:03.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:24:03.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:24:03.253 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2023-12-19T19:24:06.289 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:24:15.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:24:15.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:24:18.357 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:24:30.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:24:30.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:24:33.425 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:24:36.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:24:36.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:24:36.501 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:24:36.520 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:24:36.520 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi089 2023-12-19T19:24:36.536 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-19T19:24:36.550 INFO:teuthology.task.internal:Transferring archived files... 2023-12-19T19:24:36.551 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270/remote/smithi059 2023-12-19T19:24:36.552 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-19T19:24:36.604 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270/remote/smithi089 2023-12-19T19:24:36.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:24:36.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:24:39.665 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:24:48.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:24:48.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:24:51.729 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:25:03.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:25:03.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:25:06.801 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:25:09.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:25:09.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:25:09.873 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:25:09.874 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:25:09.874 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi089 2023-12-19T19:25:09.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-19T19:25:09.905 INFO:teuthology.task.internal:Not uploading archives. 2023-12-19T19:25:09.906 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-19T19:25:09.924 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-19T19:25:09.925 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-19T19:25:09.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:25:09.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:25:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 19 19:21 /home/ubuntu/cephtest 2023-12-19T19:25:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 19 19:24 /home/ubuntu/cephtest/archive 2023-12-19T19:25:09.948 INFO:teuthology.orchestra.run.smithi059.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 19 18:29 /home/ubuntu/cephtest/archive/syslog 2023-12-19T19:25:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: 265245 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Dec 19 18:29 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T19:25:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: 265244 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Dec 19 18:29 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T19:25:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: 402358 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 19 19:22 /home/ubuntu/cephtest/archive/audit 2023-12-19T19:25:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: 426298 292 -rw------- 1 ubuntu ubuntu 296870 Dec 19 19:22 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-12-19T19:25:09.949 INFO:teuthology.orchestra.run.smithi059.stdout: 265280 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Dec 19 18:38 /home/ubuntu/cephtest/cephadm 2023-12-19T19:25:09.950 INFO:teuthology.orchestra.run.smithi059.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-19T19:25:13.009 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:25:22.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:25:22.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:25:25.077 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:25:37.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:25:37.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:25:40.145 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:25:43.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2023-12-19T19:25:43.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:25:43.217 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-12-19T19:25:43.218 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:25:43.218 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi089 2023-12-19T19:25:43.233 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-19T19:25:43.246 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-19T19:25:43.259 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2023-12-19T19:25:43.259 INFO:teuthology.nuke:Checking targets against current locks 2023-12-19T19:25:43.289 DEBUG:teuthology.nuke:shortname: smithi059 2023-12-19T19:25:43.289 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T19:25:43.312 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 18:20:57.626385', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA3NlDAzADjs2CAiF8SUiU1CqJkoIMMlo5ctm5kltw245hoHt/gmD5NzfMizrAUe/wvfzqDuHwqvjsPPp65bHXg='} 2023-12-19T19:25:43.316 DEBUG:teuthology.nuke:shortname: smithi089 2023-12-19T19:25:43.316 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T19:25:43.336 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497270', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 18:20:57.627828', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALVpFu+Wjmuv9JQ18CMiNiRBsCf+d32vdJbhBhrXfZ4KCETRVOChoasgP7fFX4R12DLFOymDH8TxuruOPLGhx4='} 2023-12-19T19:25:43.366 INFO:teuthology.orchestra.console.smithi059:Power off 2023-12-19T19:25:43.366 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T19:25:43.390 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2023-12-19T19:25:43.390 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:25:43.407 INFO:teuthology.orchestra.console.smithi089:Power off 2023-12-19T19:25:43.407 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T19:25:43.417 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2023-12-19T19:25:43.428 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2023-12-19T19:25:43.428 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:25:43.449 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-12-19T19:25:47.419 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:25:47.451 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:25:47.529 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2023-12-19T19:25:47.561 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2023-12-19T19:25:51.533 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:25:51.562 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:25:51.645 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2023-12-19T19:25:51.673 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2023-12-19T19:25:51.746 INFO:teuthology.orchestra.console.smithi059:Power off completed 2023-12-19T19:25:51.774 INFO:teuthology.orchestra.console.smithi089:Power off completed 2023-12-19T19:25:51.880 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3183.0061807632446 failure_reason: 'Command failed on smithi059 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf854274-9e9d-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c ''ceph orch ps''"' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=63f517d953144809b3d10ecfa2be3948 status: fail success: false 2023-12-19T19:25:51.880 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T19:25:51.954 INFO:teuthology.run:FAIL